2024-03-29T11:07:46.488 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:07:46.488 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/orig.config.yaml 2024-03-29T11:07:46.526 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:46.611 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622688' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQRrBRMAfHdhGld6YwrTI+gcrcuY4ZGQ6eZPfb+SEII/W1Ud8U9x3m9CUG8aEOrXCwN0BLNEH12AXsdQvZ+lnc= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBnQWxnGUNHEHELviyjeTBjknWQL26n5SWHO5wnQbwINWwIuFxYgXK4piOozwC0FHhyobrZijV9TgSstsfCG2k= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:07:46.612 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:07:46.612 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:07:46.613 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:07:46.614 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:07:46.640 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:07:46.640 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:07:46.640 INFO:teuthology.packaging:ref: None 2024-03-29T11:07:46.640 INFO:teuthology.packaging:tag: None 2024-03-29T11:07:46.640 INFO:teuthology.packaging:branch: main 2024-03-29T11:07:46.640 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:46.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:07:47.037 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:07:47.039 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:07:47.053 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:07:47.054 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:07:47.132 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:07:47.147 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:07:47.198 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:47.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.796945', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQRrBRMAfHdhGld6YwrTI+gcrcuY4ZGQ6eZPfb+SEII/W1Ud8U9x3m9CUG8aEOrXCwN0BLNEH12AXsdQvZ+lnc='} 2024-03-29T11:07:47.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.798182', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBnQWxnGUNHEHELviyjeTBjknWQL26n5SWHO5wnQbwINWwIuFxYgXK4piOozwC0FHhyobrZijV9TgSstsfCG2k='} 2024-03-29T11:07:47.248 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:07:47.275 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:07:47.275 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:07:47.275 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:07:47.423 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f05c4f790d0>, signals=[15]) 2024-03-29T11:07:47.423 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:07:47.503 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:07:47.503 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:07:47.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:47.597 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:07:47.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:47.685 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:07:47.692 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-29T11:07:47.710 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-29T11:07:47.710 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:47.766 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:47.767 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:47.767 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:47.767 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:47.767 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-29T11:07:47.796 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-29T11:07:47.814 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-29T11:07:47.814 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-29T11:07:47.870 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-29T11:07:47.870 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-29T11:07:47.870 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:47.871 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:47.871 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-29T11:07:47.896 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:07:47.905 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:07:47.941 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:07:47.941 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:47.944 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:47.959 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:07:48.034 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:07:48.035 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:48.037 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:48.123 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:07:48.189 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:07:48.219 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:07:48.219 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:07:48.220 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:48.220 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:48.249 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-29T11:07:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:48.251 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-29T11:07:48.309 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:48.309 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-361.el9.x86_64 2024-03-29T11:07:48.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-03-29T11:07:48.314 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:48.314 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-361.el9.x86_64 2024-03-29T11:07:48.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-03-29T11:07:50.312 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:07:50.412 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:07:51.846 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-29T11:07:51.937 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:07:52.744 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-29T11:08:01.094 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 924 kB/s | 8.0 MB 00:08 2024-03-29T11:08:02.685 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-03-29T11:08:03.511 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:08:06.810 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-29T11:08:08.562 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-29T11:08:09.169 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-29T11:08:09.921 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-29T11:08:11.012 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:08:11.939 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:08:16.020 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:08:17.122 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 5.2 kB/s | 1.7 kB 00:00 2024-03-29T11:08:18.285 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:18.285 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:18.380 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:08:18.391 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:08:18.391 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-03-29T11:08:18.392 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:08:18.393 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 76 M 2024-03-29T11:08:18.393 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 123 M 2024-03-29T11:08:18.393 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:08:18.537 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:08:19.221 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T11:08:19.387 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:19.673 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:08:19.674 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:08:19.675 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:08:19.675 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:08:19.676 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:08:19.676 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:19.676 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:08:19.676 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-03-29T11:08:19.677 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:08:19.678 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 76 M 2024-03-29T11:08:19.678 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 123 M 2024-03-29T11:08:19.678 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:08:19.773 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 716 kB/s | 192 kB 00:00 2024-03-29T11:08:19.921 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T11:08:20.163 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:08:20.166 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:20.166 INFO:teuthology.orchestra.run.smithi117.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:08:20.166 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:08:20.546 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:08:20.546 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:08:20.890 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-03-29T11:08:21.380 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:08:21.380 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:08:21.557 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T11:08:22.015 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-03-29T11:08:22.018 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:22.018 INFO:teuthology.orchestra.run.smithi173.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-29T11:08:22.019 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:08:22.399 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:08:22.399 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:08:23.220 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:08:23.221 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:08:23.647 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:08:24.234 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:24.321 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:25.502 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:08:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:26.190 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:27.099 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:28.960 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:29.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:31.959 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:32.006 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:33.864 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:36.752 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:38.750 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:06.060 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:06.060 INFO:teuthology.orchestra.run.smithi117.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:06.060 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:09:06.103 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:06.429 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:06.429 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:06.429 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:06.430 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:06.805 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:06.806 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:06.806 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:09:06.806 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:09:07.152 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:07.278 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:07.279 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:07.279 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:07.279 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:07.279 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:07.279 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:07.279 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:07.279 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:07.360 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:07.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:07.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-03-29T11:09:07.394 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-03-29T11:09:07.394 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-03-29T11:09:07.394 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-03-29T11:09:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:09:07.396 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:07.397 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:07.397 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:07.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-29T11:09:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.318 INFO:teuthology.orchestra.run.smithi173.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:09:08.362 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.722 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:08.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:08.722 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:09.117 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:09:09.118 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:09:09.359 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:09.400 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-29T11:09:09.403 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T11:09:09.418 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.p7XgAfcOFh 2024-03-29T11:09:09.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p7XgAfcOFh 2024-03-29T11:09:09.445 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:09.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.p7XgAfcOFh 2024-03-29T11:09:09.551 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.552 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.552 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:09.552 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:09.553 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:09.553 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:09.553 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:09.623 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.p7XgAfcOFh is 6KB 2024-03-29T11:09:09.632 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.633 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-03-29T11:09:09.671 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.p7XgAfcOFh 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-03-29T11:09:09.676 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:09.677 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:09.678 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:09.678 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-03-29T11:09:09.678 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:09:09.678 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:09.679 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:09.679 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:09.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:09.757 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:09.757 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:09.757 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:09.757 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:09.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-29T11:09:10.349 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-03-29T11:09:11.743 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:11.787 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-03-29T11:09:11.789 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-29T11:09:11.804 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.SEKMIIFztY 2024-03-29T11:09:11.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SEKMIIFztY 2024-03-29T11:09:11.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.SEKMIIFztY 2024-03-29T11:09:12.015 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.SEKMIIFztY is 6KB 2024-03-29T11:09:12.063 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.SEKMIIFztY 2024-03-29T11:09:12.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:12.143 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:12.143 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:12.143 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:12.144 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:12.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:12.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-03-29T11:09:39.946 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:09:39.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:09:39.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:42.302 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:09:42.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:09:42.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:58.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T11:10:00.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-29T11:10:07.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:10:07.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:09.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:10:09.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:10.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T11:10:12.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-29T11:10:22.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:10:22.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:24.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:10:24.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:40.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-03-29T11:10:55.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:10:55.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:56.336 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-29T11:10:56.627 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-29T11:10:56.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:10:56.777 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:56.778 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-29T11:10:56.793 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:56.793 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:56.793 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:56.793 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:57.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:10:57.050 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:57.050 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-29T11:10:57.065 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:57.065 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:57.065 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:57.065 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:57.794 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:10:57.795 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:57.872 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-03-29T11:10:57.907 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:58.066 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:10:58.067 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:58.141 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-03-29T11:10:58.175 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:58.176 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:10:58.184 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:10:58.184 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:58.186 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:58.200 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:10:58.270 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:10:58.312 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:10:58.312 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:58.315 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:58.348 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:10:58.377 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:10:58.378 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:58.384 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:58.415 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:58.424 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:58.428 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:58.437 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:58.439 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:10:58.450 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:10:58.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:58.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:58.505 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:10:58.516 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:10:58.517 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:58.532 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:58.560 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:58.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:58.675 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:58.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:58.794 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:10:58.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:58.862 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:58.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:58.972 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:59.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:59.061 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:10:59.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:59.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-29T11:10:59.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-29T11:10:59.172 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:59.413 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:59.768 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:10:59.777 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:10:59.777 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:10:59.837 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:10:59.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-29T11:10:59.960 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-29T11:10:59.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-29T11:11:00.041 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-29T11:11:00.044 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:00.045 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-29T11:11:00.089 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-29T11:11:00.089 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-29T11:11:00.128 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-29T11:11:00.128 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:00.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:00.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:00.213 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:00.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:00.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:00.355 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:00.366 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:00.383 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:00.384 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:25.311 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-29T11:15:25.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:15:25.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:25.412 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-29T11:15:25.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-29T11:15:25.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:15:25.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:25.586 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-29T11:15:25.624 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T11:15:25.625 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:25.636 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:25.636 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:25.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:25.639 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:25.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:25.668 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:25.685 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:25.695 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:25.712 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:25.714 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-29T11:15:25.728 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-29T11:15:25.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-29T11:15:25.745 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:25.755 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-29T11:15:25.761 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:25.771 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:25.787 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:25.818 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:25.821 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:25.821 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-29T11:15:25.822 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.822 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.822 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.822 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.838 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:25.842 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:25.842 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-29T11:15:25.842 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.842 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.842 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.843 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:25.843 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:25.851 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:25.851 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:25.851 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:25.851 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:25.857 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:25.857 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:25.857 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:25.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:25.860 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:25.860 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:26.006 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:26.006 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:26.029 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:26.029 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:26.120 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:26.121 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:15:26.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:26.136 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:26.136 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:15:26.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:26.148 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:26.149 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:26.167 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:26.167 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:26.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:26.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:26.296 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:26.317 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:26.362 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-29T11:15:26.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-29T11:15:26.383 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-29T11:15:26.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-29T11:15:26.611 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-29T11:15:26.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:26.643 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-29T11:15:26.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:27.343 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 201 kB/s | 82 kB 00:00 2024-03-29T11:15:27.434 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 179 kB/s | 82 kB 00:00 2024-03-29T11:15:27.785 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T11:15:27.806 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T11:15:28.194 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T11:15:28.215 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T11:15:28.613 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:15:28.659 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:15:30.060 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:15:30.155 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-29T11:15:30.958 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:15:32.179 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-29T11:15:35.743 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-03-29T11:15:39.727 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T11:15:42.178 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T11:15:43.699 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T11:15:48.671 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T11:15:49.593 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-29T11:15:51.272 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:15:51.283 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:15:51.284 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:15:51.285 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-03-29T11:15:51.286 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:15:51.287 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-03-29T11:15:51.287 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:15:52.806 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-03-29T11:15:52.865 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T11:15:53.031 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.3 MB/s | 742 kB 00:00 2024-03-29T11:15:53.182 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.4 MB/s | 9.7 MB 00:01 2024-03-29T11:15:53.207 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.8 MB/s | 498 kB 00:00 2024-03-29T11:15:53.291 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 542 kB/s | 45 kB 00:00 2024-03-29T11:15:53.391 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-29T11:15:53.500 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.5 MB/s | 162 kB 00:00 2024-03-29T11:15:53.608 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.9 MB/s | 320 kB 00:00 2024-03-29T11:15:53.709 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T11:15:53.792 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:15:53.859 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 7.5 MB/s | 5.1 MB 00:00 2024-03-29T11:15:53.976 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-29T11:15:53.985 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 6.1 MB 00:00 2024-03-29T11:15:54.001 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T11:15:54.018 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:15:54.085 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-29T11:15:54.110 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-29T11:15:54.127 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-29T11:15:54.153 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:15:54.170 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 834 kB/s | 35 kB 00:00 2024-03-29T11:15:54.186 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:15:54.412 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T11:15:54.454 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-03-29T11:15:54.504 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2024-03-29T11:15:54.563 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:15:54.630 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T11:15:54.697 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-29T11:15:54.722 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T11:15:54.814 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-29T11:15:55.148 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 3.4 MB 00:00 2024-03-29T11:15:55.215 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.8 MB/s | 3.1 MB 00:00 2024-03-29T11:15:55.619 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T11:15:56.779 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T11:16:00.523 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 20 MB 00:09 2024-03-29T11:16:00.528 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:00.528 INFO:teuthology.orchestra.run.smithi173.stdout:Total 6.3 MB/s | 58 MB 00:09 2024-03-29T11:16:00.528 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:16:00.566 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:16:00.566 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:16:01.031 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:16:01.031 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:16:01.787 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-29T11:16:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:16:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:02.754 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:02.978 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:03.032 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:03.371 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:03.422 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:03.708 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:03.890 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:03.925 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:04.036 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:04.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:04.117 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:04.150 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:04.179 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:04.211 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:04.432 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:16:04.444 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:04.444 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:16:04.444 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:04.445 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:04.446 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:04.447 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-03-29T11:16:04.448 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-03-29T11:16:04.448 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:04.448 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-03-29T11:16:04.448 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:16:04.533 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:04.558 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:04.665 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:04.696 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:04.741 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:05.196 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:05.228 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:05.282 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:05.350 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:05.380 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:05.688 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.6 MB/s | 5.3 MB 00:01 2024-03-29T11:16:05.755 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 379 kB/s | 25 kB 00:00 2024-03-29T11:16:05.913 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.6 MB/s | 742 kB 00:00 2024-03-29T11:16:06.038 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.9 MB/s | 498 kB 00:00 2024-03-29T11:16:06.172 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 9.7 MB 00:01 2024-03-29T11:16:06.231 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 774 kB/s | 45 kB 00:00 2024-03-29T11:16:06.306 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T11:16:06.406 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T11:16:06.507 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.1 MB/s | 320 kB 00:00 2024-03-29T11:16:06.615 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.7 MB/s | 298 kB 00:00 2024-03-29T11:16:06.707 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:16:06.816 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 5.1 MB 00:00 2024-03-29T11:16:06.874 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 633 kB/s | 106 kB 00:00 2024-03-29T11:16:06.900 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:16:06.916 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:16:07.000 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-29T11:16:07.025 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-29T11:16:07.050 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T11:16:07.067 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:16:07.143 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 379 kB/s | 35 kB 00:00 2024-03-29T11:16:07.168 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 461 kB/s | 46 kB 00:00 2024-03-29T11:16:07.352 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 7.1 MB/s | 20 MB 00:02 2024-03-29T11:16:07.519 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:07.543 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 181 kB/s | 26 kB 00:00 2024-03-29T11:16:07.569 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 726 kB/s | 309 kB 00:00 2024-03-29T11:16:07.577 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:07.594 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-29T11:16:07.619 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T11:16:07.736 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-29T11:16:07.770 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-29T11:16:07.795 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:16:07.896 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-29T11:16:08.134 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:08.171 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.4 MB/s | 3.1 MB 00:00 2024-03-29T11:16:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:08.262 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:08.263 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:08.596 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.2 MB/s | 3.4 MB 00:00 2024-03-29T11:16:08.601 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:08.601 INFO:teuthology.orchestra.run.smithi117.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T11:16:08.602 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:16:08.635 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:16:08.635 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:16:09.110 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:16:09.111 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:16:10.588 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:16:10.673 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:10.700 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:10.725 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:10.757 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:10.774 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:10.957 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:11.012 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:11.381 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:11.427 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:11.502 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:11.589 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:11.612 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:11.658 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:11.686 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:11.722 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:11.789 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:11.814 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:11.861 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:11.896 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:12.003 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:12.039 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:12.079 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:12.111 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:12.140 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:12.173 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:12.490 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:15.481 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:15.529 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:16.080 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:16.115 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:16.207 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:16.208 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:16.208 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:16.208 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:16.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:17.663 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:17.663 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-29T11:16:17.663 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-29T11:16:17.663 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-29T11:16:17.664 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-29T11:16:17.664 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-29T11:16:17.664 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-29T11:16:17.664 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-29T11:16:17.664 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:18.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:18.612 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:18.614 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:18.688 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:18.688 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:21.038 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:21.039 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:21.040 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:21.041 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:16:21.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-03-29T11:16:21.880 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:33 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:16:22.031 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:22.032 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-29T11:16:22.033 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:22.034 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 48 M 2024-03-29T11:16:22.034 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 204 M 2024-03-29T11:16:22.034 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:16:22.250 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:16:22.300 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:16:22.334 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:16:22.493 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:16:22.885 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 207 kB/s | 163 kB 00:00 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-29T11:16:25.941 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-29T11:16:25.942 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-29T11:16:25.942 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-29T11:16:25.942 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:26.850 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:26.874 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:26.874 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:26.874 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:26.874 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:26.874 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:26.875 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:26.876 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:26.953 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:26.953 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:28.085 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:28.086 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:28.087 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:29.080 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:16:29.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-03-29T11:16:30.020 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:16:30.175 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:30.176 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-03-29T11:16:30.178 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 204 M 2024-03-29T11:16:30.178 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:16:30.368 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:16:30.435 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-29T11:16:30.468 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:16:30.627 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 396 kB/s | 163 kB 00:00 2024-03-29T11:16:30.677 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 897 kB/s | 187 kB 00:00 2024-03-29T11:16:40.018 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 4.8 MB/s | 47 MB 00:09 2024-03-29T11:16:40.020 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:40.021 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.9 MB/s | 48 MB 00:09 2024-03-29T11:16:40.033 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:16:40.052 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:16:40.053 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:16:40.428 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:16:40.429 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:16:40.999 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:16:41.076 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:16:41.115 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:16:41.147 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:41.198 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:16:41.221 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:45.681 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:45.763 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:16:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:16:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:16:48.080 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:16:48.489 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:16:48.490 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:16:48.490 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:48.490 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:16:48.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-03-29T11:16:49.335 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:48 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:16:49.486 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:16:49.489 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:16:49.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:16:49.491 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout:Install 36 Packages 2024-03-29T11:16:49.492 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:16:49.493 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 30 M 2024-03-29T11:16:49.493 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 107 M 2024-03-29T11:16:49.493 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:16:50.062 INFO:teuthology.orchestra.run.smithi117.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:16:50.307 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 47 MB 00:28 2024-03-29T11:16:50.309 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:50.310 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.7 MB/s | 48 MB 00:28 2024-03-29T11:16:50.324 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:16:50.345 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:16:50.345 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:16:50.722 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:16:50.722 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:16:51.304 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:16:51.374 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:16:51.419 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:16:51.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:51.495 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:16:51.518 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:52.853 INFO:teuthology.orchestra.run.smithi117.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 474 kB/s | 1.5 MB 00:03 2024-03-29T11:16:56.002 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:16:56.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:57.011 INFO:teuthology.orchestra.run.smithi117.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 295 kB/s | 2.1 MB 00:07 2024-03-29T11:16:57.611 INFO:teuthology.orchestra.run.smithi117.stdout:(4/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 17 MB 00:04 2024-03-29T11:16:57.678 INFO:teuthology.orchestra.run.smithi117.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 371 kB/s | 247 kB 00:00 2024-03-29T11:16:57.745 INFO:teuthology.orchestra.run.smithi117.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:16:57.770 INFO:teuthology.orchestra.run.smithi117.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-29T11:16:57.787 INFO:teuthology.orchestra.run.smithi117.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 761 kB/s | 32 kB 00:00 2024-03-29T11:16:57.804 INFO:teuthology.orchestra.run.smithi117.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:16:57.837 INFO:teuthology.orchestra.run.smithi117.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-29T11:16:57.871 INFO:teuthology.orchestra.run.smithi117.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-29T11:16:58.005 INFO:teuthology.orchestra.run.smithi117.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-03-29T11:16:58.030 INFO:teuthology.orchestra.run.smithi117.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 649 kB/s | 103 kB 00:00 2024-03-29T11:16:58.064 INFO:teuthology.orchestra.run.smithi117.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-29T11:16:58.089 INFO:teuthology.orchestra.run.smithi117.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T11:16:58.189 INFO:teuthology.orchestra.run.smithi117.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T11:16:58.214 INFO:teuthology.orchestra.run.smithi117.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-29T11:16:58.273 INFO:teuthology.orchestra.run.smithi117.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-29T11:16:58.315 INFO:teuthology.orchestra.run.smithi117.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-03-29T11:16:58.340 INFO:teuthology.orchestra.run.smithi117.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T11:16:58.374 INFO:teuthology.orchestra.run.smithi117.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-03-29T11:16:58.407 INFO:teuthology.orchestra.run.smithi117.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 531 kB/s | 18 kB 00:00 2024-03-29T11:16:58.432 INFO:teuthology.orchestra.run.smithi117.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T11:16:58.458 INFO:teuthology.orchestra.run.smithi117.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-03-29T11:16:58.483 INFO:teuthology.orchestra.run.smithi117.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-29T11:16:58.508 INFO:teuthology.orchestra.run.smithi117.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2024-03-29T11:16:58.530 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:16:58.530 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:16:58.530 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:16:58.530 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:58.530 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:16:58.531 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:16:58.533 INFO:teuthology.orchestra.run.smithi117.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-03-29T11:16:58.559 INFO:teuthology.orchestra.run.smithi117.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:16:58.584 INFO:teuthology.orchestra.run.smithi117.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T11:16:58.609 INFO:teuthology.orchestra.run.smithi117.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-29T11:16:58.635 INFO:teuthology.orchestra.run.smithi117.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T11:16:58.660 INFO:teuthology.orchestra.run.smithi117.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-03-29T11:16:58.685 INFO:teuthology.orchestra.run.smithi117.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:16:58.711 INFO:teuthology.orchestra.run.smithi117.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-03-29T11:16:58.728 INFO:teuthology.orchestra.run.smithi117.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 476 kB/s | 20 kB 00:00 2024-03-29T11:16:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:16:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:16:58.937 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:16:59.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-03-29T11:16:59.860 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:00.013 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:00.016 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:00.017 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:00.018 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout:Install 36 Packages 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:00.019 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-03-29T11:17:00.020 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 107 M 2024-03-29T11:17:00.020 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:00.852 INFO:teuthology.orchestra.run.smithi173.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-29T11:17:01.061 INFO:teuthology.orchestra.run.smithi117.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 436 kB/s | 4.7 MB 00:10 2024-03-29T11:17:01.064 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:01.065 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.6 MB/s | 30 MB 00:11 2024-03-29T11:17:01.088 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:01.130 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:01.130 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:01.556 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:01.556 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 997 kB/s | 1.5 MB 00:01 2024-03-29T11:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-29T11:17:02.777 INFO:teuthology.orchestra.run.smithi173.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 928 kB/s | 247 kB 00:00 2024-03-29T11:17:02.894 INFO:teuthology.orchestra.run.smithi173.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:17:02.928 INFO:teuthology.orchestra.run.smithi173.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.3 MB/s | 144 kB 00:00 2024-03-29T11:17:02.961 INFO:teuthology.orchestra.run.smithi173.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T11:17:02.987 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:02.995 INFO:teuthology.orchestra.run.smithi173.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:03.028 INFO:teuthology.orchestra.run.smithi173.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:17:03.055 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:03.070 INFO:teuthology.orchestra.run.smithi173.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:17:03.096 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:03.123 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:03.137 INFO:teuthology.orchestra.run.smithi173.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:17:03.171 INFO:teuthology.orchestra.run.smithi173.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:17:03.195 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:03.205 INFO:teuthology.orchestra.run.smithi173.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:17:03.238 INFO:teuthology.orchestra.run.smithi173.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:17:03.239 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:03.272 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:03.365 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:03.414 INFO:teuthology.orchestra.run.smithi173.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 714 kB/s | 125 kB 00:00 2024-03-29T11:17:03.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:03.508 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:03.539 INFO:teuthology.orchestra.run.smithi173.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-29T11:17:03.581 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-29T11:17:03.634 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-29T11:17:03.672 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-03-29T11:17:03.730 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 8.4 MB/s | 358 kB 00:00 2024-03-29T11:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-03-29T11:17:03.785 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 529 kB/s | 18 kB 00:00 2024-03-29T11:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 700 kB/s | 23 kB 00:00 2024-03-29T11:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 782 kB/s | 19 kB 00:00 2024-03-29T11:17:03.891 INFO:teuthology.orchestra.run.smithi173.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 796 kB/s | 20 kB 00:00 2024-03-29T11:17:03.917 INFO:teuthology.orchestra.run.smithi173.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-03-29T11:17:03.950 INFO:teuthology.orchestra.run.smithi173.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 2.3 MB/s | 79 kB 00:00 2024-03-29T11:17:03.972 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:03.992 INFO:teuthology.orchestra.run.smithi173.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-03-29T11:17:04.018 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:04.026 INFO:teuthology.orchestra.run.smithi173.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 491 kB/s | 16 kB 00:00 2024-03-29T11:17:04.048 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:04.059 INFO:teuthology.orchestra.run.smithi173.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.7 MB/s | 90 kB 00:00 2024-03-29T11:17:04.083 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:04.093 INFO:teuthology.orchestra.run.smithi173.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-03-29T11:17:04.113 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:04.127 INFO:teuthology.orchestra.run.smithi173.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-03-29T11:17:04.169 INFO:teuthology.orchestra.run.smithi173.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 10 MB/s | 427 kB 00:00 2024-03-29T11:17:04.190 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:04.202 INFO:teuthology.orchestra.run.smithi173.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 603 kB/s | 20 kB 00:00 2024-03-29T11:17:04.223 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:04.257 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:04.298 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:04.332 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:04.367 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:04.457 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:04.497 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:04.571 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:04.614 INFO:teuthology.orchestra.run.smithi173.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 4.7 MB 00:03 2024-03-29T11:17:04.717 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:05.016 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:05.044 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:05.069 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:05.069 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:05.069 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:05.069 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:05.070 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:05.070 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:06.503 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:06.529 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:06.529 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:06.529 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:06.529 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:06.529 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:06.530 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:06.970 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:06.994 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:06.995 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:06.995 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:06.995 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:06.995 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:07.196 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:07.220 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:07.278 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:08.929 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:08.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:08.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:08.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.855 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:09.856 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:09.857 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:09.858 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:09.858 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:09.858 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:10.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-03-29T11:17:10.244 INFO:teuthology.orchestra.run.smithi173.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 17 MB 00:08 2024-03-29T11:17:10.248 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:10.248 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.9 MB/s | 30 MB 00:10 2024-03-29T11:17:10.273 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:10.317 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:10.317 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:10.733 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:10.734 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:10.774 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:10.872 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:10.925 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:10.926 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:10.926 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:10.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-03-29T11:17:11.559 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:11.709 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:11.710 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 747 k 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 754 k 2024-03-29T11:17:11.711 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:12.046 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:12.105 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:12.146 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:12.174 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:12.245 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:12.314 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:12.329 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.2 MB/s | 747 kB 00:00 2024-03-29T11:17:12.330 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:12.330 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.2 MB/s | 747 kB 00:00 2024-03-29T11:17:12.330 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:12.335 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:12.335 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:12.341 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:12.341 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:12.347 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:12.418 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:12.438 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:12.508 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:12.676 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:12.714 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:12.762 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:12.818 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:13.002 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:13.051 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:13.090 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:13.124 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:13.154 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:13.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:13.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:13.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:13.424 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:13.513 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:13.565 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:13.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:13.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:13.889 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:14.047 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:14.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:14.102 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:14.370 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:14.522 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:14.526 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:17:14.527 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 145 k 2024-03-29T11:17:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 436 k 2024-03-29T11:17:14.527 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:14.960 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 333 kB/s | 145 kB 00:00 2024-03-29T11:17:14.961 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:14.961 INFO:teuthology.orchestra.run.smithi117.stdout:Total 332 kB/s | 145 kB 00:00 2024-03-29T11:17:14.961 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:14.969 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:14.969 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:15.020 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:15.021 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:15.157 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:15.211 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:15.236 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:15.236 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:15.236 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:15.236 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:16.025 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:16.026 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:16.026 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:16.026 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:16.026 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:16.026 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:16.177 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:16.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:16.177 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:16.177 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:16.178 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:16.178 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:16.237 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:16.260 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:16.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-03-29T11:17:17.027 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:17.124 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:17.177 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:17.177 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:17.177 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:17.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:17.813 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:17.963 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:17.965 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-29T11:17:17.966 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.9 M 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-29T11:17:17.967 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:18.184 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:18.185 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:18.186 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:18.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:18.830 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 153 kB/s | 31 kB 00:00 2024-03-29T11:17:18.856 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 67 kB/s | 15 kB 00:00 2024-03-29T11:17:19.023 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-29T11:17:19.041 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:19.041 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:19.041 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:19.043 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:19.044 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:19.064 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-29T11:17:19.173 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-03-29T11:17:19.257 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 234 kB/s | 19 kB 00:00 2024-03-29T11:17:19.282 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-03-29T11:17:19.382 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-29T11:17:19.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-03-29T11:17:19.432 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 624 kB/s | 31 kB 00:00 2024-03-29T11:17:19.475 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-03-29T11:17:19.500 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:17:19.675 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.7 MB/s | 1.8 MB 00:01 2024-03-29T11:17:19.679 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:19.679 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.0 MB/s | 6.9 MB 00:01 2024-03-29T11:17:19.696 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:19.712 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:19.712 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:19.881 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:19.882 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:20.151 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:20.152 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:17:20.152 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:20.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-03-29T11:17:20.388 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:20.459 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:20.502 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:20.541 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:20.571 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:20.672 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:20.798 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:20.900 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:20.936 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 747 k 2024-03-29T11:17:20.937 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 754 k 2024-03-29T11:17:20.938 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:20.942 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:20.970 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:21.002 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:21.269 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:21.628 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.1 MB/s | 747 kB 00:00 2024-03-29T11:17:21.629 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:21.629 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.1 MB/s | 747 kB 00:00 2024-03-29T11:17:21.629 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:21.635 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:21.635 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:21.641 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:21.641 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:21.718 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:22.025 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:22.026 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:22.171 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:22.257 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:22.356 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:22.357 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:22.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:22.660 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:22.932 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:23.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:23.173 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:23.323 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:23.324 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:23.325 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:23.326 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 37 M 2024-03-29T11:17:23.327 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 209 M 2024-03-29T11:17:23.327 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:23.713 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:23.731 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-03-29T11:17:23.773 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-29T11:17:23.857 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:17:23.864 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:23.865 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:23.865 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T11:17:23.865 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 436 k 2024-03-29T11:17:23.866 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:23.882 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 318 kB/s | 34 kB 00:00 2024-03-29T11:17:24.099 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:17:24.166 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:17:24.191 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:17:24.359 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-29T11:17:24.392 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 6.2 MB/s | 195 kB 00:00 2024-03-29T11:17:24.409 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 268 kB/s | 145 kB 00:00 2024-03-29T11:17:24.409 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:24.409 INFO:teuthology.orchestra.run.smithi173.stdout:Total 267 kB/s | 145 kB 00:00 2024-03-29T11:17:24.410 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:24.418 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:24.418 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:24.471 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:24.471 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:24.601 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-29T11:17:24.628 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:24.684 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.7 MB/s | 7.4 MB 00:01 2024-03-29T11:17:24.685 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:24.709 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:24.709 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:24.709 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:24.709 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:25.535 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 215 kB/s | 245 kB 00:01 2024-03-29T11:17:25.538 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:25.540 INFO:teuthology.orchestra.run.smithi117.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:17:25.583 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:25.609 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:25.609 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:25.699 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:25.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-03-29T11:17:25.974 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:25.974 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:26.517 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:26.616 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:26.669 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:26.670 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:17:26.670 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:26.679 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:26.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:26.757 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:26.788 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:26.881 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:26.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:27.110 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:27.132 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:27.218 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:27.312 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:27.464 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:27.465 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-29T11:17:27.466 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:27.467 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.9 M 2024-03-29T11:17:27.467 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-29T11:17:27.467 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:27.863 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:27.977 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:28.479 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 58 kB/s | 31 kB 00:00 2024-03-29T11:17:28.530 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 26 kB/s | 15 kB 00:00 2024-03-29T11:17:28.672 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:17:28.722 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-03-29T11:17:29.207 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:29.405 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 802 kB/s | 548 kB 00:00 2024-03-29T11:17:29.497 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-29T11:17:29.822 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.3 MB/s | 2.0 MB 00:00 2024-03-29T11:17:29.881 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 939 kB/s | 1.8 MB 00:01 2024-03-29T11:17:29.931 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:17:30.032 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T11:17:30.132 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-03-29T11:17:30.237 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:30.265 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:30.599 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 448 kB/s | 863 kB 00:01 2024-03-29T11:17:30.604 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:30.604 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.2 MB/s | 6.9 MB 00:03 2024-03-29T11:17:30.624 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:30.642 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:30.642 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:30.828 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:30.828 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:31.348 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:31.662 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:31.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:31.872 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:31.916 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:31.944 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:31.976 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:32.209 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:32.243 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:32.739 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:32.739 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:32.740 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:33.007 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:33.008 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:33.224 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:33.224 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:33.224 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:33.224 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:33.225 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:33.346 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:33.347 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:33.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:33.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:34.176 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:34.192 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:34.326 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:34.327 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:34.328 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-03-29T11:17:34.329 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:34.330 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.0 M 2024-03-29T11:17:34.330 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-29T11:17:34.330 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:34.344 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:34.345 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T11:17:34.345 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:34.345 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:34.346 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-03-29T11:17:34.347 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:34.589 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 238 kB/s | 24 kB 00:00 2024-03-29T11:17:34.622 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 273 kB/s | 32 kB 00:00 2024-03-29T11:17:34.631 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 18 kB 00:00 2024-03-29T11:17:34.680 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 499 kB/s | 17 kB 00:00 2024-03-29T11:17:34.681 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:17:34.707 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:17:34.714 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:17:34.732 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:17:34.757 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-29T11:17:34.881 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:17:34.991 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 104 kB/s | 52 kB 00:00 2024-03-29T11:17:35.023 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-29T11:17:35.033 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T11:17:35.058 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-29T11:17:35.090 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-29T11:17:35.115 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:17:35.292 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 674 kB/s | 203 kB 00:00 2024-03-29T11:17:35.400 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-29T11:17:35.609 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 283 kB/s | 90 kB 00:00 2024-03-29T11:17:35.643 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-29T11:17:35.646 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:35.646 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:17:35.647 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:35.659 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:35.659 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:35.682 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-29T11:17:35.749 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:35.749 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:36.287 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:36.349 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 293 kB/s | 195 kB 00:00 2024-03-29T11:17:36.449 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-29T11:17:36.453 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:36.501 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:36.557 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:36.594 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:36.635 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:36.728 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:36.799 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 477 kB/s | 803 kB 00:01 2024-03-29T11:17:36.828 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:36.878 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:37.588 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:37.634 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:37.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:37.754 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:37.779 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:38.534 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:38.534 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:38.534 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:38.535 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:38.896 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:38.897 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:39.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:17:39.718 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:39.871 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:17:39.872 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:39.873 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.8 M 2024-03-29T11:17:39.874 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-03-29T11:17:39.874 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:40.462 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:17:40.487 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 846 kB/s | 248 kB 00:00 2024-03-29T11:17:40.554 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 867 kB/s | 58 kB 00:00 2024-03-29T11:17:40.596 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-29T11:17:40.621 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-03-29T11:17:40.946 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 183 kB/s | 138 kB 00:00 2024-03-29T11:17:40.947 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:40.947 INFO:teuthology.orchestra.run.smithi117.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-29T11:17:40.961 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:40.971 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:40.971 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:41.016 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.1 MB/s | 7.4 MB 00:06 2024-03-29T11:17:41.019 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:41.019 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.6 MB/s | 37 MB 00:06 2024-03-29T11:17:41.059 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:41.059 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:41.063 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:41.088 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:41.088 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:41.362 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:41.439 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:41.440 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:41.532 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:17:41.590 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:17:42.126 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:17:42.136 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:42.210 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:17:42.236 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:42.271 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:17:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:42.328 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:42.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:42.550 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:42.580 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:42.657 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:43.070 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:43.071 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:17:43.071 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:17:43.071 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:17:43.071 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:17:43.071 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:17:43.320 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:43.377 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:17:43.377 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:43.377 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:43.378 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:43.434 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:43.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-03-29T11:17:44.184 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:44.333 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:44.334 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:44.334 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 964 k 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.8 M 2024-03-29T11:17:44.335 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:44.464 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 613 kB/s | 79 kB 00:00 2024-03-29T11:17:44.692 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:45.139 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-29T11:17:45.140 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:45.141 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-29T11:17:45.141 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:45.161 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:45.161 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:45.235 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:45.235 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:45.420 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:45.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:17:45.632 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:45.754 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:46.304 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:17:46.304 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:17:46.602 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:17:46.602 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:46.602 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:46.602 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:17:46.603 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:46.603 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:46.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-03-29T11:17:47.453 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:47.608 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:47.608 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-29T11:17:47.609 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:47.610 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 332 k 2024-03-29T11:17:47.610 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.5 M 2024-03-29T11:17:47.610 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:47.737 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 69 kB 00:00 2024-03-29T11:17:48.220 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 432 kB/s | 264 kB 00:00 2024-03-29T11:17:48.221 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:48.222 INFO:teuthology.orchestra.run.smithi117.stdout:Total 543 kB/s | 332 kB 00:00 2024-03-29T11:17:48.222 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:48.230 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:48.230 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:48.266 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:48.267 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:48.412 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:48.589 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:17:48.623 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:48.640 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:17:48.640 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:48.640 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:17:48.640 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:48.737 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:48.738 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:48.739 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:48.739 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:48.739 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:48.739 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:49.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:49.415 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:17:49.728 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:49.729 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:49.730 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:49.881 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:49.882 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-03-29T11:17:49.883 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:49.884 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-03-29T11:17:49.884 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-29T11:17:49.884 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:50.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-03-29T11:17:50.371 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2024-03-29T11:17:50.396 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T11:17:50.438 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:17:50.454 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-29T11:17:50.488 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:17:50.514 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 730 kB/s | 43 kB 00:00 2024-03-29T11:17:50.626 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:50.689 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 119 kB/s | 52 kB 00:00 2024-03-29T11:17:50.776 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:50.777 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-03-29T11:17:50.778 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 456 k 2024-03-29T11:17:50.778 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:50.797 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-03-29T11:17:50.823 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 48 kB/s | 15 kB 00:00 2024-03-29T11:17:50.989 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 356 kB/s | 59 kB 00:00 2024-03-29T11:17:51.106 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 487 kB/s | 203 kB 00:00 2024-03-29T11:17:51.132 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-29T11:17:51.165 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-29T11:17:51.169 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:51.169 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:17:51.170 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:51.182 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:51.182 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:51.251 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 267 kB/s | 127 kB 00:00 2024-03-29T11:17:51.252 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:51.252 INFO:teuthology.orchestra.run.smithi117.stdout:Total 267 kB/s | 127 kB 00:00 2024-03-29T11:17:51.252 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:51.257 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:51.258 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:51.273 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:51.273 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:51.312 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:51.313 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:51.457 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:51.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:51.791 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:52.003 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:52.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:52.087 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:52.117 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:52.158 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:52.252 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:52.300 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:52.300 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:52.300 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:52.300 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.300 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:52.301 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:52.360 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:52.409 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:52.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-03-29T11:17:53.133 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:53.182 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:53.216 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:53.233 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:53.264 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:53.286 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:53.290 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:53.292 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:53.292 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:53.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-03-29T11:17:53.929 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:54.081 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:54.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:54.082 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T11:17:54.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:54.082 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 31 k 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 140 k 2024-03-29T11:17:54.083 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:54.134 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:54.135 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:54.135 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:54.135 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:54.135 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:54.246 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 192 kB/s | 31 kB 00:00 2024-03-29T11:17:54.246 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:54.246 INFO:teuthology.orchestra.run.smithi117.stdout:Total 190 kB/s | 31 kB 00:00 2024-03-29T11:17:54.247 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:17:54.250 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:17:54.250 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:17:54.283 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:17:54.285 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:17:54.414 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:17:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:54.489 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:54.490 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:54.520 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:54.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:17:54.861 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:17:55.133 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:55.321 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:55.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-03-29T11:17:55.474 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:55.474 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:55.474 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-29T11:17:55.476 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:55.476 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.8 M 2024-03-29T11:17:55.476 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-03-29T11:17:55.476 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:55.902 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:55.950 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:17:55.998 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:56.008 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 747 kB/s | 248 kB 00:00 2024-03-29T11:17:56.050 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:56.050 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 369 kB/s | 138 kB 00:00 2024-03-29T11:17:56.051 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:56.051 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:56.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-03-29T11:17:56.134 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-03-29T11:17:56.159 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 469 kB/s | 51 kB 00:00 2024-03-29T11:17:56.193 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-03-29T11:17:56.195 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.196 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2024-03-29T11:17:56.209 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:17:56.220 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:17:56.220 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:17:56.308 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:17:56.309 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:17:56.644 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:17:56.666 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:56.762 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:56.808 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:17:56.814 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:56.815 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:56.815 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:56.852 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:17:56.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-03-29T11:17:57.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:17:57.427 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:57.447 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:17:57.508 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:17:57.525 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:57.531 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:57.576 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:57.577 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:57.577 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:57.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-03-29T11:17:58.207 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:58.305 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:58.357 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:58.358 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:58.358 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:58.397 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:17:58.397 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:17:58.397 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:17:58.398 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:17:58.398 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:17:58.398 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:17:58.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-03-29T11:17:58.711 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:17:58.711 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:58.711 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:17:58.711 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:58.712 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:17:58.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-03-29T11:17:58.979 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:59.081 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:59.135 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:59.136 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:59.136 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:59.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-03-29T11:17:59.535 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:17:59.686 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:17:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 964 k 2024-03-29T11:17:59.688 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.8 M 2024-03-29T11:17:59.688 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:17:59.751 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:17:59.811 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 637 kB/s | 79 kB 00:00 2024-03-29T11:17:59.849 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:59.901 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:17:59.902 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-29T11:17:59.902 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:17:59.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-03-29T11:18:00.078 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 885 kB 00:00 2024-03-29T11:18:00.079 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:00.080 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 964 kB 00:00 2024-03-29T11:18:00.080 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:00.100 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:00.100 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:00.174 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:00.174 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:00.355 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:00.489 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:00.507 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:18:00.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:00.655 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:18:00.656 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:00.656 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:00.656 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:00.656 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 231 k 2024-03-29T11:18:00.657 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:18:00.882 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 378 kB/s | 85 kB 00:00 2024-03-29T11:18:00.882 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:00.883 INFO:teuthology.orchestra.run.smithi117.stdout:Total 376 kB/s | 85 kB 00:00 2024-03-29T11:18:00.883 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:18:00.890 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:18:00.890 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:18:00.945 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:18:00.945 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:18:01.078 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:18:01.180 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:01.232 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:01.232 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:01.519 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:01.556 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:01.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-03-29T11:18:01.842 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:01.842 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:01.842 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:18:01.843 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:01.843 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:01.843 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:18:02.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-03-29T11:18:02.376 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:02.531 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T11:18:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 332 k 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.5 M 2024-03-29T11:18:02.533 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:02.634 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:18:02.652 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 580 kB/s | 69 kB 00:00 2024-03-29T11:18:02.785 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T11:18:02.786 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:02.787 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T11:18:02.787 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:02.790 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:02.791 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:18:02.792 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:02.792 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-03-29T11:18:02.792 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 13 M 2024-03-29T11:18:02.792 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:18:02.795 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:02.795 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:02.833 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:02.833 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:02.979 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:03.205 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:03.205 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:03.205 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:03.205 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:03.410 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:18:03.411 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:03.411 INFO:teuthology.orchestra.run.smithi117.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:18:03.411 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:18:03.420 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:18:03.420 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:18:03.495 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:18:03.495 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:18:03.932 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:18:03.986 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:03.996 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:04.011 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:04.011 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:04.011 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:04.011 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:04.012 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:04.012 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:04.366 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:04.367 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:04.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-03-29T11:18:05.104 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:05.104 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:05.105 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:18:05.105 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:05.105 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:05.105 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:18:05.283 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:05.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-03-29T11:18:05.434 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:05.434 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:05.434 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 127 k 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-03-29T11:18:05.435 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:05.926 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:16:01 AM UTC. 2024-03-29T11:18:05.958 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 242 kB/s | 127 kB 00:00 2024-03-29T11:18:05.958 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:05.958 INFO:teuthology.orchestra.run.smithi173.stdout:Total 242 kB/s | 127 kB 00:00 2024-03-29T11:18:05.959 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:05.965 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:05.965 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:06.022 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:06.022 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:06.074 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-29T11:18:06.074 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:06.074 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:06.075 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 171 k 2024-03-29T11:18:06.076 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 494 k 2024-03-29T11:18:06.076 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-29T11:18:06.167 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:06.290 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:06.737 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 259 kB/s | 171 kB 00:00 2024-03-29T11:18:06.738 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:06.738 INFO:teuthology.orchestra.run.smithi117.stdout:Total 259 kB/s | 171 kB 00:00 2024-03-29T11:18:06.739 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-29T11:18:06.747 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-29T11:18:06.747 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-29T11:18:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:06.807 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-29T11:18:06.807 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-29T11:18:06.954 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-29T11:18:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:07.050 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:07.050 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:07.051 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:07.051 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:07.062 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-03-29T11:18:07.510 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.814 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:07.815 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:07.815 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-29T11:18:07.815 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:07.815 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:07.815 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-29T11:18:07.917 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:08.018 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:08.076 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:08.077 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:08.077 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:08.077 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:08.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:08.729 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:08.882 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:08.884 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:08.884 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T11:18:08.884 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:08.884 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:08.884 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:08.885 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 k 2024-03-29T11:18:08.886 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 140 k 2024-03-29T11:18:08.886 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:09.294 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 76 kB/s | 31 kB 00:00 2024-03-29T11:18:09.294 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:09.295 INFO:teuthology.orchestra.run.smithi173.stdout:Total 76 kB/s | 31 kB 00:00 2024-03-29T11:18:09.295 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:09.298 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:09.298 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:09.328 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:09.329 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:09.459 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:09.601 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:10.265 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:10.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-03-29T11:18:11.002 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:11.100 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:11.151 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:11.152 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:11.152 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:11.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-03-29T11:18:11.762 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:11.859 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:11.911 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:11.912 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:11.912 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:11.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-03-29T11:18:12.533 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:12.634 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:12.686 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:12.687 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:12.688 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:12.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-03-29T11:18:13.311 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:13.412 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:13.465 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:13.466 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:13.466 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:13.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-03-29T11:18:14.090 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:14.189 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:14.242 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:14.243 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:14.243 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:14.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-03-29T11:18:14.867 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:14.966 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:15.018 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:15.019 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T11:18:15.020 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:15.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-03-29T11:18:15.638 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:15.789 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:15.789 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-03-29T11:18:15.790 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:16.130 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 251 kB/s | 85 kB 00:00 2024-03-29T11:18:16.130 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:16.131 INFO:teuthology.orchestra.run.smithi173.stdout:Total 250 kB/s | 85 kB 00:00 2024-03-29T11:18:16.131 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:16.138 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:16.138 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:16.193 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:16.193 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:16.428 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:16.810 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:17.130 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:17.131 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:17.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-03-29T11:18:17.937 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:18.090 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:18.091 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:18.092 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-03-29T11:18:18.093 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:18.474 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-03-29T11:18:18.475 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:18.475 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-29T11:18:18.476 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:18.484 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:18.484 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:18.558 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:18.558 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:18.983 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:19.042 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:19.066 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:19.066 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:19.066 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:19.067 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:19.067 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:19.067 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:20.227 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:20.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-03-29T11:18:21.039 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:15:48 AM UTC. 2024-03-29T11:18:21.189 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T11:18:21.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:21.191 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 171 k 2024-03-29T11:18:21.191 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-03-29T11:18:21.191 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T11:18:21.419 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 752 kB/s | 171 kB 00:00 2024-03-29T11:18:21.420 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:21.420 INFO:teuthology.orchestra.run.smithi173.stdout:Total 748 kB/s | 171 kB 00:00 2024-03-29T11:18:21.420 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T11:18:21.428 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T11:18:21.428 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T11:18:21.484 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T11:18:21.484 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T11:18:21.630 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T11:18:21.725 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.132 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.421 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.422 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:22.422 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T11:18:22.422 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:22.422 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:22.422 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T11:18:22.649 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:22.649 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:22.868 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:22.892 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:22.893 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:22.893 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:22.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:23.127 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:23.148 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:23.148 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:23.149 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:23.150 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:23.150 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:23.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:23.184 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:23.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:23.216 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:23.216 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:23.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:23.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:23.326 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:23.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:23.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:23.419 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:23.420 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:23.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:23.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:23.522 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:23.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:23.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:23.618 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:23.619 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:23.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:23.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:23.720 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:23.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:23.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:23.814 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:23.902 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:23.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:23.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:23.932 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:23.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:23.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:24.003 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:24.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:24.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:24.083 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:24.083 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:24.083 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622688', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQRrBRMAfHdhGld6YwrTI+gcrcuY4ZGQ6eZPfb+SEII/W1Ud8U9x3m9CUG8aEOrXCwN0BLNEH12AXsdQvZ+lnc=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBnQWxnGUNHEHELviyjeTBjknWQL26n5SWHO5wnQbwINWwIuFxYgXK4piOozwC0FHhyobrZijV9TgSstsfCG2k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:24.083 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:24.114 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:24.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:24.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:24.196 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:24.197 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:24.251 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:24.252 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:24.306 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:24.306 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:24.306 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 11:18:04.262424891 +0000 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 11:15:03.453176007 +0000 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 11:15:03.453176007 +0000 2024-03-29T11:18:24.307 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 11:15:03.453176007 +0000 2024-03-29T11:18:24.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:24.372 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T11:18:24.373 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T11:18:24.373 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000231928 s, 2.2 MB/s 2024-03-29T11:18:24.374 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:24.429 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:24.484 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 11:18:04.262424891 +0000 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 11:15:03.810169118 +0000 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 11:15:03.810169118 +0000 2024-03-29T11:18:24.485 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 11:15:03.810169118 +0000 2024-03-29T11:18:24.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:24.550 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T11:18:24.550 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T11:18:24.550 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000203857 s, 2.5 MB/s 2024-03-29T11:18:24.551 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:24.607 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:24.663 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:24.663 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:24.663 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:18:24.663 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:24.663 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:24.664 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 11:18:04.263424872 +0000 2024-03-29T11:18:24.664 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 11:15:04.156162441 +0000 2024-03-29T11:18:24.664 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 11:15:04.156162441 +0000 2024-03-29T11:18:24.664 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 11:15:04.156162441 +0000 2024-03-29T11:18:24.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:24.727 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T11:18:24.727 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T11:18:24.727 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000173903 s, 2.9 MB/s 2024-03-29T11:18:24.729 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:24.784 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:24.838 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:24.838 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:24.838 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-29 11:18:04.263424872 +0000 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-29 11:15:04.497155861 +0000 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-29 11:15:04.497155861 +0000 2024-03-29T11:18:24.839 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-29 11:15:04.497155861 +0000 2024-03-29T11:18:24.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:24.904 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-29T11:18:24.904 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-29T11:18:24.904 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000234612 s, 2.2 MB/s 2024-03-29T11:18:24.905 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:24.961 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:24.961 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:24.961 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:24.977 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:24.977 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 11:18:19.354009115 +0000 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 11:14:52.964014063 +0000 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 11:14:52.964014063 +0000 2024-03-29T11:18:25.033 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 11:14:52.964014063 +0000 2024-03-29T11:18:25.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:25.100 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T11:18:25.100 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T11:18:25.100 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000270977 s, 1.9 MB/s 2024-03-29T11:18:25.102 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:25.159 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:25.214 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:25.214 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 11:18:19.354009115 +0000 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 11:14:53.498002002 +0000 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 11:14:53.498002002 +0000 2024-03-29T11:18:25.215 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 11:14:53.498002002 +0000 2024-03-29T11:18:25.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:25.281 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T11:18:25.281 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T11:18:25.281 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000229295 s, 2.2 MB/s 2024-03-29T11:18:25.282 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:25.339 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 11:18:19.355009093 +0000 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 11:14:53.845994141 +0000 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 11:14:53.845994141 +0000 2024-03-29T11:18:25.394 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 11:14:53.845994141 +0000 2024-03-29T11:18:25.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T11:18:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T11:18:25.460 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000178217 s, 2.9 MB/s 2024-03-29T11:18:25.462 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:25.518 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 11:18:19.355009093 +0000 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 11:14:54.187986417 +0000 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 11:14:54.187986417 +0000 2024-03-29T11:18:25.573 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 11:14:54.187986417 +0000 2024-03-29T11:18:25.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:25.638 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T11:18:25.639 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T11:18:25.639 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000207549 s, 2.5 MB/s 2024-03-29T11:18:25.640 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:25.696 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:25.696 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:25.696 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:25.698 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:25.698 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:25.698 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:25.698 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:25.698 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:25.698 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:25.698 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:25.698 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:25.699 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:25.699 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:25.699 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:18:25.699 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:25.699 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:25.699 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:25.700 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:25.701 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:25.701 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:25.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:25.758 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:25.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:25.805 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:25.831 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.117:6789]'), ('mon.c', '[v1:172.21.15.117:6790]'), ('mon.b', '[v1:172.21.15.173:6789]')] 2024-03-29T11:18:25.832 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.173:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:25.832 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:25.832 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:25.886 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.117:6789]' --addv c '[v1:172.21.15.117:6790]' --addv b '[v1:172.21.15.173:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:25.971 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:25.971 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:18:25.971 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-03-29T11:18:25.971 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-03-29T11:18:25.971 INFO:teuthology.orchestra.run.smithi117.stdout:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:18:25.972 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:25.973 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:25.988 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5b002eed-53a1-4f21-85da-94f1de320a49... 2024-03-29T11:18:25.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:26.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-03-29T11:18:26.077 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:26.078 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.079 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.173:6789]" 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: lockdep = True 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 1 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: ms bind msgr2 = False 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-03-29T11:18:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: ms type = async 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:26.081 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-03-29T11:18:26.082 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 20 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 20 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:26.083 INFO:teuthology.orchestra.run.smithi117.stdout: osd class default list = * 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd class load list = * 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd client watch timeout = 120 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-03-29T11:18:26.084 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:26.085 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:26.086 INFO:teuthology.orchestra.run.smithi117.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: paxos service trim min = 10 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-29T11:18:26.087 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 20 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 20 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-03-29T11:18:26.088 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:26.089 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.090 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "[v1:172.21.15.117:6789],[v1:172.21.15.117:6790],[v1:172.21.15.173:6789]" 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: lockdep = True 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 1 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: ms bind msgr2 = False 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 5000 2024-03-29T11:18:26.091 INFO:teuthology.orchestra.run.smithi173.stdout: ms type = async 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:26.092 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-03-29T11:18:26.093 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore zero block detection = True 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:26.094 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd class default list = * 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd class load list = * 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd client watch timeout = 120 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T11:18:26.095 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:26.096 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:26.097 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: paxos service trim min = 10 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2024-03-29T11:18:26.098 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T11:18:26.099 INFO:teuthology.orchestra.run.smithi173.stdout: debug objecter = 20 2024-03-29T11:18:26.099 INFO:teuthology.orchestra.run.smithi173.stdout: debug rados = 20 2024-03-29T11:18:26.099 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-03-29T11:18:26.099 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2024-03-29T11:18:26.099 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2024-03-29T11:18:26.100 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:26.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:26.156 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:26.156 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:26.156 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:26.170 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:26.171 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:26.225 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:18:26.225 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:26.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:26.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:26.305 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:26.305 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:26.359 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:18:26.360 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:26.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:26.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:26.399 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:26.399 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:26.454 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:26.454 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:26.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:26.509 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:26.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:26.582 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:26.584 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:26.584 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:26.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:26.627 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:26.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:26.684 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:26.699 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:26.700 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:26.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:26.727 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:26.727 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:26.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:18:26.727 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:26.801 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:27.310 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T11:18:27.312 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T11:18:27.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:27.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:27.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:27.578 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:27.578 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:27.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:18:27.579 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:27.654 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:27.654 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:27.654 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:27.654 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:27.654 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:27.655 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:27.655 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:27.655 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:27.655 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:27.655 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:28.097 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T11:18:28.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T11:18:28.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:28.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:28.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:28.291 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:28.291 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:28.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:18:28.292 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:28.366 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:28.366 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:28.366 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:28.367 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:28.848 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T11:18:28.850 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T11:18:28.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:29.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:29.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:29.099 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:29.099 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:29.100 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:18:29.100 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:29.174 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:29.175 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:29.684 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-29T11:18:29.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-29T11:18:29.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:29.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:29.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:29.999+0000 7fa20d945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:29.999+0000 7fa20d945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:29.999+0000 7fa20d945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:29.999+0000 7fa20d945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:29.999+0000 7fa20d945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:29.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:30.000+0000 7fa20d945640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:18:32.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:32.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:32.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.966+0000 7fc73c237640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:18:32.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.966+0000 7fc73c237640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:18:32.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.966+0000 7fc73c237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:32.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.966+0000 7fc73c237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:32.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.966+0000 7fc73c237640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:32.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:32.967+0000 7fc73c237640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:18:35.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:35.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:35.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.942+0000 7f5317573640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:18:35.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.942+0000 7f5317573640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:18:35.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.942+0000 7f5317573640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:35.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.942+0000 7f5317573640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:35.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.942+0000 7f5317573640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:35.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:35.943+0000 7f5317573640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:18:38.771 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:38.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:38.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.918+0000 7fbaba813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:18:38.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.918+0000 7fbaba813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:18:38.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.918+0000 7fbaba813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:38.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.918+0000 7fbaba813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:38.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.918+0000 7fbaba813640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:38.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:38.919+0000 7fbaba813640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:18:41.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:41.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:41.807 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:41.808 INFO:tasks.ceph:role: osd.4 2024-03-29T11:18:41.808 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:18:41.808 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:41.887 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:41.888 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:42.399 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T11:18:42.401 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T11:18:42.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:42.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:42.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:42.640 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:42.640 INFO:tasks.ceph:role: osd.5 2024-03-29T11:18:42.640 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:18:42.640 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:42.715 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:43.164 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T11:18:43.166 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T11:18:43.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:43.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:43.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:43.358 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:43.358 INFO:tasks.ceph:role: osd.6 2024-03-29T11:18:43.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:18:43.358 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:43.432 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:43.433 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:43.919 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T11:18:43.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T11:18:43.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:44.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:44.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:44.173 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:44.173 INFO:tasks.ceph:role: osd.7 2024-03-29T11:18:44.173 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:18:44.173 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:44.248 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:44.249 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:44.249 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:44.249 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:44.767 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T11:18:44.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T11:18:44.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:44.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:44.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.076+0000 7fd8e579a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.076+0000 7fd8e579a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.076+0000 7fd8e579a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.076+0000 7fd8e579a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.076+0000 7fd8e579a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:18:45.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:45.077+0000 7fd8e579a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:18:47.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:47.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.049+0000 7f586980d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.049+0000 7f586980d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.049+0000 7f586980d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.049+0000 7f586980d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.049+0000 7f586980d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:18:48.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:48.050+0000 7f586980d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:18:50.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:18:50.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:50.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:18:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:18:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:18:51.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:51.023+0000 7f56b5391640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:18:53.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:18:53.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:18:53.987+0000 7fa1be68b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:18:56.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:18:56.843 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:18:56.844 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:56.844 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:18:56.876 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:56.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:18:56.946 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:56.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:18:57.015 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:57.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:18:57.083 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:57.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:18:57.152 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:18:57.183 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:18:57.253 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:18:57.322 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:18:57.391 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:18:57.459 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:18:57.459 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:18:57.475 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:18:57.475 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:18:57.514 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:18:57.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:18:57.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:18:57.558 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCCowZm7rTLHxAAGMW23WdevtJSckc5CQ7ECw== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCGowZmdiMLABAAbEBqHd5lO0H071bwIWS+CQ== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCIowZmJ7WoORAAQlFaRVMHPDBZIW3mgF51IQ== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCLowZmz548OBAA+XWfuwpo9G9BkKTAYN9rpw== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCOowZmeyPINhAAgwkE3wL11BQjgzgzFLrZ9w== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCCowZmd9MkJBAAzTXbE7mSRa3RYZ1AAuM8KA== 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-03-29T11:18:57.559 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCVowZmRLugBBAAylhHVAT+kGUYqLR5ByVByQ== 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCYowZm5sD5AhAACekO3ld1zj2u2pfnO5sGPA== 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCbowZmKRNlARAAkkkBzf8gAMzeOGezRDJK7g== 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCdowZmpcjcOhAAuaVWqbAcYjfdN1wEiFqUNw== 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCCowZm4dHYJhAAaHy4j3MOiMoGXRcsgjDy2A== 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout:[client.1] 2024-03-29T11:18:57.560 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCCowZmJYBCKhAAYs5midBhz6viX+Wo0wR8Nw== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.y] 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCCowZm7rTLHxAAGMW23WdevtJSckc5CQ7ECw== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCGowZmdiMLABAAbEBqHd5lO0H071bwIWS+CQ== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCIowZmJ7WoORAAQlFaRVMHPDBZIW3mgF51IQ== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCLowZmz548OBAA+XWfuwpo9G9BkKTAYN9rpw== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCOowZmeyPINhAAgwkE3wL11BQjgzgzFLrZ9w== 2024-03-29T11:18:57.582 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCCowZmd9MkJBAAzTXbE7mSRa3RYZ1AAuM8KA== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCVowZmRLugBBAAylhHVAT+kGUYqLR5ByVByQ== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCYowZm5sD5AhAACekO3ld1zj2u2pfnO5sGPA== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.6] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCbowZmKRNlARAAkkkBzf8gAMzeOGezRDJK7g== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.7] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCdowZmpcjcOhAAuaVWqbAcYjfdN1wEiFqUNw== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCCowZm4dHYJhAAaHy4j3MOiMoGXRcsgjDy2A== 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-03-29T11:18:57.583 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCCowZmJYBCKhAAYs5midBhz6viX+Wo0wR8Nw== 2024-03-29T11:18:57.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:57.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:57.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:57.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:18:57.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:57.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:18:58.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:58.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:58.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:58.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:18:58.273 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:18:58.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:18:58.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:58.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:18:58.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:18:58.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:58.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:18:58.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:18:58.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:58.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:18:58.762 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:58.765 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:58.818 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:18:58.818 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:18:58.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:18:58.820 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:18:58.820 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:18:58.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:18:58.822 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:18:58.822 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:18:58.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:18:58.861 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:18:58.861 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:18:58.861 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:18:58.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:18:58.863 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:18:58.864 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:18:58.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:18:58.865 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:18:58.866 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:18:58.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:18:58.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.990+0000 7f7bc380f640 10 monclient: get_monmap_and_config 2024-03-29T11:18:58.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.990+0000 7f7bc380f640 10 monclient: build_initial_monmap 2024-03-29T11:18:58.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.990+0000 7f7bc380f640 10 monclient: monmap: 2024-03-29T11:18:58.967 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T11:18:58.967 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:58.991396+0000 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:58.991396+0000 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:18:58.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.990+0000 7f7bc380f640 1 Processor -- start 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 1 -- start start 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 10 monclient: init 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-b con 0x7f7bbc09ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.991+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-c con 0x7f7bbc09c870 addr v1:172.21.15.173:6789/0 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-a con 0x7f7bbc0b4360 addr v1:172.21.15.117:6789/0 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc09dec0 con 0x7f7bbc0b4360 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc09da30 con 0x7f7bbc09c870 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc0b39c0 con 0x7f7bbc09ac40 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:58.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc380f640 10 monclient(hunting): authenticate will time out at 807.301087s 2024-03-29T11:18:58.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc15ab640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7bbc09ac40 0x7f7bbc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34910/0 (socket says 172.21.15.117:34910) 2024-03-29T11:18:58.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:18:58.992+0000 7f7bc15ab640 1 -- 172.21.15.117:0/2207496731 learned_addr learned my addr 172.21.15.117:0/2207496731 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): tick 2024-03-29T11:19:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:01.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.117:6790/0 conn(0x7f7bbc09ac40 legacy=0x7f7bbc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.173:6789/0 conn(0x7f7bbc09c870 legacy=0x7f7bbc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.117:6789/0 conn(0x7f7bbc0b4360 legacy=0x7f7bbc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba00036e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba0007300 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba000b020 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc0b5160 con 0x7f7ba00036e0 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba000fdf0 con 0x7f7ba0007300 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 1 -- 172.21.15.117:0/2207496731 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba0011030 con 0x7f7ba000b020 2024-03-29T11:19:01.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:01.992+0000 7f7baeffd640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720661299 0 0) 0x7f7bbc0b5160 con 0x7f7ba00036e0 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:04.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.164+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba8003520 con 0x7f7ba00036e0 2024-03-29T11:19:04.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128033361 0 0) 0x7f7ba8003520 con 0x7f7ba00036e0 2024-03-29T11:19:04.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.117:6790/0 conn(0x7f7ba000b020 legacy=0x7f7ba000d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.173:6789/0 conn(0x7f7ba0007300 legacy=0x7f7ba0009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 monclient: found mon.noname-a 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba0012290 con 0x7f7ba00036e0 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb00016d0 con 0x7f7ba00036e0 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: dump: 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.144 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7bc380f640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7bc380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bb00018a0 con 0x7f7ba00036e0 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7bc380f640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 1 -- 172.21.15.117:0/2207496731 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb0001c30 con 0x7f7ba00036e0 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7bc380f640 4 set_mon_vals no callback set 2024-03-29T11:19:04.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.166+0000 7f7baffff640 10 monclient: dump: 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.146 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.167+0000 7f7bc380f640 10 monclient: shutdown 2024-03-29T11:19:04.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 1 -- 172.21.15.117:0/2207496731 >> v1:172.21.15.117:6789/0 conn(0x7f7ba00036e0 legacy=0x7f7ba0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 1 -- 172.21.15.117:0/2207496731 shutdown_connections 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 1 -- 172.21.15.117:0/2207496731 >> 172.21.15.117:0/2207496731 conn(0x7f7bbc066b00 msgr2=0x7f7bbc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 1 -- 172.21.15.117:0/2207496731 shutdown_connections 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.168+0000 7f7bc380f640 1 -- 172.21.15.117:0/2207496731 wait complete. 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 10 monclient: monmap: 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:04.152 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 librados: starting msgr at 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 librados: starting objecter 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 Processor -- start 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 -- start start 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 librados: setting wanted keys 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 1 librados: calling monclient init 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.169+0000 7f7bc380f640 10 monclient: init 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba00036e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba0007300 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc15ab640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7ba00036e0 0x7f7bbc1dae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34960/0 (socket says 172.21.15.117:34960) 2024-03-29T11:19:04.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba000b020 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc15ab640 1 -- 172.21.15.117:0/821413226 learned_addr learned my addr 172.21.15.117:0/821413226 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc1da6b0 con 0x7f7ba000b020 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc1da990 con 0x7f7ba0007300 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bbc145010 con 0x7f7ba00036e0 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7bc380f640 10 monclient(hunting): authenticate will time out at 812.479817s 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7badffb640 10 client.?.objecter ms_handle_connect 0x7f7ba00036e0 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7badffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.170+0000 7f7badffb640 10 client.?.objecter ms_handle_connect 0x7f7ba000b020 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 client.?.objecter ms_handle_connect 0x7f7ba0007300 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651221460 0 0) 0x7f7bbc1da6b0 con 0x7f7ba000b020 2024-03-29T11:19:04.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba0005e00 con 0x7f7ba000b020 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537175330 0 0) 0x7f7bbc1da990 con 0x7f7ba0007300 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bbc1da6b0 con 0x7f7ba0007300 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728850988 0 0) 0x7f7bbc145010 con 0x7f7ba00036e0 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bbc1da990 con 0x7f7ba00036e0 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201491420 0 0) 0x7f7ba0005e00 con 0x7f7ba000b020 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bbc145010 con 0x7f7ba000b020 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282315110 0 0) 0x7f7bbc1da6b0 con 0x7f7ba0007300 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba0005e00 con 0x7f7ba0007300 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb4001900 con 0x7f7ba000b020 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): dump: 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.156 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb8001820 con 0x7f7ba0007300 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): dump: 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.157 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662126195 0 0) 0x7f7bbc1da990 con 0x7f7ba00036e0 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bbc1da6b0 con 0x7f7ba00036e0 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb0001bb0 con 0x7f7ba00036e0 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient(hunting): dump: 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.158 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4144745114 0 0) 0x7f7bbc145010 con 0x7f7ba000b020 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 >> v1:172.21.15.117:6790/0 conn(0x7f7ba00036e0 legacy=0x7f7bbc1dae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 >> v1:172.21.15.173:6789/0 conn(0x7f7ba0007300 legacy=0x7f7bbc1453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.171+0000 7f7badffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 monclient: found mon.a 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bbc1db530 con 0x7f7ba000b020 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 10 monclient: _renew_subs 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc142ff0 con 0x7f7ba000b020 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bb4001a60 con 0x7f7ba000b020 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7bb4002090 con 0x7f7ba000b020 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 monclient: dump: 2024-03-29T11:19:04.160 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7baf7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 10 monclient: _renew_subs 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7bb4003d80 con 0x7f7ba000b020 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc1464d0 con 0x7f7ba000b020 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7bc380f640 1 librados: init done 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4012099501 0 0) 0x7f7bb4004030 con 0x7f7ba000b020 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 10 client.4113.objecter ms_dispatch 0x7f7bbc000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.172+0000 7f7badffb640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.173+0000 7f7bc380f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.173+0000 7f7bc380f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.173+0000 7f7bc380f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.173+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc09b020 con 0x7f7ba000b020 2024-03-29T11:19:04.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.175+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7bb4004580 con 0x7f7ba000b020 2024-03-29T11:19:04.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.175+0000 7f7badffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.175+0000 7f7badffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:04.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.243+0000 7f7bc380f640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:04.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.243+0000 7f7bc380f640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:04.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.243+0000 7f7bc380f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:04.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.243+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7bbc13d340 con 0x7f7ba000b020 2024-03-29T11:19:04.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.244+0000 7f7badffb640 1 -- 172.21.15.117:0/821413226 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f7bb4004900 con 0x7f7ba000b020 2024-03-29T11:19:04.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.244+0000 7f7badffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:04.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.244+0000 7f7badffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:04.222 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-03-29T11:19:04.223 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.245+0000 7f7bc380f640 10 librados: watch_flush enter 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 10 librados: watch_flush exit 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 10 monclient: shutdown 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 >> v1:172.21.15.117:6789/0 conn(0x7f7ba000b020 legacy=0x7f7bbc145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 shutdown_connections 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 >> 172.21.15.117:0/821413226 conn(0x7f7bbc066b00 msgr2=0x7f7bbc13c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 shutdown_connections 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 -- 172.21.15.117:0/821413226 wait complete. 2024-03-29T11:19:04.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:04.246+0000 7f7bc380f640 1 librados: shutdown 2024-03-29T11:19:04.234 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:04.234 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:04.234 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:19:04.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:04.262 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:19:04.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:04.329 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:19:04.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:04.397 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-29T11:19:04.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:04.463 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:19:04.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:04.491 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:19:04.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:04.558 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:19:04.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:04.623 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:04.646+0000 7f16bc410040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:04.625 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T11:19:04.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:04.636 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:04.659+0000 7f7664248040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:04.679 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:04.702+0000 7f16bc410040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:04.689 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:04.712+0000 7f7664248040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:04.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 7bc368d8-fe29-40fa-bf23-810bd2452126 0 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.854+0000 7f9a291c6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.854+0000 7f9a291c6640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.854+0000 7f9a291c6640 10 monclient: monmap: 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:04.856235+0000 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:04.856235+0000 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.855+0000 7f9a291c6640 1 Processor -- start 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.855+0000 7f9a291c6640 1 -- start start 2024-03-29T11:19:04.833 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.855+0000 7f9a291c6640 10 monclient: init 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.855+0000 7f9a291c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2409ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2409c870 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a240b4360 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2409dec0 con 0x7f9a240b4360 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2409da30 con 0x7f9a2409ac40 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a240b39c0 con 0x7f9a2409c870 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient(hunting): authenticate will time out at 814.001684s 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a22d9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9a2409ac40 0x7f9a240b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56554/0 (socket says 172.21.15.173:56554) 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a22d9d640 1 -- 172.21.15.173:0/2162525638 learned_addr learned my addr 172.21.15.173:0/2162525638 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:04.842 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a2259c640 1 -- 172.21.15.173:0/2162525638 >> v1:172.21.15.117:6790/0 conn(0x7f9a2409c870 legacy=0x7f9a2413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a2259c640 1 -- 172.21.15.173:0/2162525638 >> v1:172.21.15.117:6790/0 conn(0x7f9a2409c870 legacy=0x7f9a2413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a2259c640 1 --1- 172.21.15.173:0/2162525638 >> v1:172.21.15.117:6790/0 conn(0x7f9a2409c870 0x7f9a2413f3a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433518725 0 0) 0x7f9a2409da30 con 0x7f9a2409ac40 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c003520 con 0x7f9a2409ac40 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734249734 0 0) 0x7f9a1c001490 con 0x7f9a240b4360 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2409da30 con 0x7f9a240b4360 2024-03-29T11:19:04.843 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1099434624 0 0) 0x7f9a0c003520 con 0x7f9a2409ac40 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 >> v1:172.21.15.117:6790/0 conn(0x7f9a2409c870 legacy=0x7f9a2413f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 >> v1:172.21.15.117:6789/0 conn(0x7f9a240b4360 legacy=0x7f9a2414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a240b5160 con 0x7f9a2409ac40 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a18001690 con 0x7f9a2409ac40 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: dump: 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.844 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a18001870 con 0x7f9a2409ac40 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 1 -- 172.21.15.173:0/2162525638 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a18001c80 con 0x7f9a2409ac40 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a291c6640 4 set_mon_vals no callback set 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.856+0000 7f9a21d9b640 10 monclient: dump: 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 10 monclient: shutdown 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 1 -- 172.21.15.173:0/2162525638 >> v1:172.21.15.173:6789/0 conn(0x7f9a2409ac40 legacy=0x7f9a240b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 1 -- 172.21.15.173:0/2162525638 shutdown_connections 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 1 -- 172.21.15.173:0/2162525638 >> 172.21.15.173:0/2162525638 conn(0x7f9a24066b00 msgr2=0x7f9a24066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 1 -- 172.21.15.173:0/2162525638 shutdown_connections 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.857+0000 7f9a291c6640 1 -- 172.21.15.173:0/2162525638 wait complete. 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 10 monclient: monmap: 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 librados: starting msgr at 2024-03-29T11:19:04.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 librados: starting objecter 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 Processor -- start 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 -- start start 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 librados: setting wanted keys 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 1 librados: calling monclient init 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.858+0000 7f9a291c6640 10 monclient: init 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2409ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a240b12b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a241414d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a241db2c0 con 0x7f9a241414d0 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a24141960 con 0x7f9a240b12b0 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a241dadc0 con 0x7f9a2409ac40 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a291c6640 10 monclient(hunting): authenticate will time out at 814.004901s 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a2259c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9a240b12b0 0x7f9a24140dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56560/0 (socket says 172.21.15.173:56560) 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a2259c640 1 -- 172.21.15.173:0/1704340293 learned_addr learned my addr 172.21.15.173:0/1704340293 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a240b12b0 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a2409ac40 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a241414d0 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346297327 0 0) 0x7f9a24141960 con 0x7f9a240b12b0 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10004eb0 con 0x7f9a240b12b0 2024-03-29T11:19:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929235396 0 0) 0x7f9a241dadc0 con 0x7f9a2409ac40 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a24141960 con 0x7f9a2409ac40 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759600368 0 0) 0x7f9a241db2c0 con 0x7f9a241414d0 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a241dadc0 con 0x7f9a241414d0 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 337270936 0 0) 0x7f9a24141960 con 0x7f9a2409ac40 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a241db2c0 con 0x7f9a2409ac40 2024-03-29T11:19:04.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.859+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1898639872 0 0) 0x7f9a10004eb0 con 0x7f9a240b12b0 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a24141960 con 0x7f9a240b12b0 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4025339707 0 0) 0x7f9a241dadc0 con 0x7f9a241414d0 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a10004eb0 con 0x7f9a241414d0 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a18001b80 con 0x7f9a2409ac40 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.851 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a14004020 con 0x7f9a240b12b0 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.852 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a1c001b00 con 0x7f9a241414d0 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.853 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3361071658 0 0) 0x7f9a241db2c0 con 0x7f9a2409ac40 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 >> v1:172.21.15.173:6789/0 conn(0x7f9a240b12b0 legacy=0x7f9a24140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 >> v1:172.21.15.117:6789/0 conn(0x7f9a241414d0 legacy=0x7f9a241e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 monclient: found mon.c 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a241daf50 con 0x7f9a2409ac40 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 5 monclient: authenticate success, global_id 4109 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 10 monclient: _renew_subs 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a241e30a0 con 0x7f9a2409ac40 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 10 monclient: _renew_subs 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a241e3410 con 0x7f9a2409ac40 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a291c6640 1 librados: init done 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a18004bf0 con 0x7f9a2409ac40 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9a18004f00 con 0x7f9a2409ac40 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a2159a640 4 set_mon_vals no callback set 2024-03-29T11:19:04.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 monclient: dump: 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9a180053f0 con 0x7f9a2409ac40 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2391244859 0 0) 0x7f9a18006730 con 0x7f9a2409ac40 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 10 client.4109.objecter ms_dispatch 0x7f9a24000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:04.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 3 client.4109.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 3 client.4109.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.860+0000 7f9a0b7fe640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.861+0000 7f9a291c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.861+0000 7f9a291c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.861+0000 7f9a291c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.861+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a04005080 con 0x7f9a2409ac40 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.863+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9a18006a10 con 0x7f9a2409ac40 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.863+0000 7f9a0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.863+0000 7f9a0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:04.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.934+0000 7f9a291c6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7bc368d8-fe29-40fa-bf23-810bd2452126", "id": 0}] 2024-03-29T11:19:04.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.934+0000 7f9a291c6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7bc368d8-fe29-40fa-bf23-810bd2452126", "id": 0}] 2024-03-29T11:19:04.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.934+0000 7f9a291c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:04.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.934+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd new", "uuid": "7bc368d8-fe29-40fa-bf23-810bd2452126", "id": 0} v 0) v1 -- 0x7f9a040058b0 con 0x7f9a2409ac40 2024-03-29T11:19:04.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.937+0000 7f9a0b7fe640 1 -- 172.21.15.173:0/1704340293 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7bc368d8-fe29-40fa-bf23-810bd2452126", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3654086490 0 732707515) 0x7f9a18005770 con 0x7f9a2409ac40 2024-03-29T11:19:04.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.937+0000 7f9a0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7bc368d8-fe29-40fa-bf23-810bd2452126", "id": 0}] 2024-03-29T11:19:04.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.937+0000 7f9a0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:04.915 INFO:teuthology.orchestra.run.smithi173.stdout:0 2024-03-29T11:19:04.916 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 10 librados: watch_flush enter 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 10 librados: watch_flush exit 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 10 client.4109.objecter successfully canceled tick 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 10 monclient: shutdown 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 >> v1:172.21.15.117:6790/0 conn(0x7f9a2409ac40 legacy=0x7f9a241da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 shutdown_connections 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 >> 172.21.15.173:0/1704340293 conn(0x7f9a24066b00 msgr2=0x7f9a2413cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 shutdown_connections 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 -- 172.21.15.173:0/1704340293 wait complete. 2024-03-29T11:19:04.917 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:04.938+0000 7f9a291c6640 1 librados: shutdown 2024-03-29T11:19:04.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new bfb91a80-d14b-490a-9f7d-4d1e06a97587 1 2024-03-29T11:19:05.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.048+0000 7f0c581f6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.025 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.048+0000 7f0c581f6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.048+0000 7f0c581f6640 10 monclient: monmap: 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:05.049737+0000 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:05.049737+0000 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.026 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.048+0000 7f0c581f6640 1 Processor -- start 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 1 -- start start 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient: init 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5009ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5009c870 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c500b4360 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5009dec0 con 0x7f0c500b4360 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5009da30 con 0x7f0c5009ac40 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c500b39c0 con 0x7f0c5009c870 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c581f6640 10 monclient(hunting): authenticate will time out at 814.195331s 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c55f92640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0c5009ac40 0x7f0c500b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56572/0 (socket says 172.21.15.173:56572) 2024-03-29T11:19:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.049+0000 7f0c55f92640 1 -- 172.21.15.173:0/1329203742 learned_addr learned my addr 172.21.15.173:0/1329203742 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527064284 0 0) 0x7f0c5009da30 con 0x7f0c5009ac40 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c3c003520 con 0x7f0c5009ac40 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571861112 0 0) 0x7f0c48001490 con 0x7f0c500b4360 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c5009da30 con 0x7f0c500b4360 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106533493 0 0) 0x7f0c500b39c0 con 0x7f0c5009c870 2024-03-29T11:19:05.037 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c48001490 con 0x7f0c5009c870 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588401179 0 0) 0x7f0c3c003520 con 0x7f0c5009ac40 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 >> v1:172.21.15.117:6790/0 conn(0x7f0c5009c870 legacy=0x7f0c5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 >> v1:172.21.15.117:6789/0 conn(0x7f0c500b4360 legacy=0x7f0c5014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 monclient: found mon.noname-c 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c500b5160 con 0x7f0c5009ac40 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c4c001690 con 0x7f0c5009ac40 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c581f6640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c581f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: dump: 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c581f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c4c001870 con 0x7f0c5009ac40 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 1 -- 172.21.15.173:0/1329203742 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c4c001c80 con 0x7f0c5009ac40 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c581f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c581f6640 4 set_mon_vals no callback set 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.050+0000 7f0c54f90640 10 monclient: dump: 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.040 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 10 monclient: shutdown 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 1 -- 172.21.15.173:0/1329203742 >> v1:172.21.15.173:6789/0 conn(0x7f0c5009ac40 legacy=0x7f0c500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 1 -- 172.21.15.173:0/1329203742 shutdown_connections 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 1 -- 172.21.15.173:0/1329203742 >> 172.21.15.173:0/1329203742 conn(0x7f0c50066b00 msgr2=0x7f0c50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 1 -- 172.21.15.173:0/1329203742 shutdown_connections 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 1 -- 172.21.15.173:0/1329203742 wait complete. 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.051+0000 7f0c581f6640 10 monclient: monmap: 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.041 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 librados: starting msgr at 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 librados: starting objecter 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 Processor -- start 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 -- start start 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 librados: setting wanted keys 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 librados: calling monclient init 2024-03-29T11:19:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient: init 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5009ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0c500b1860 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5013b5b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c501d7660 con 0x7f0c5013b5b0 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c500ade20 con 0x7f0c5009ac40 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c501406b0 con 0x7f0c500b1860 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c581f6640 10 monclient(hunting): authenticate will time out at 814.198295s 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c55f92640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0c5009ac40 0x7f0c5013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56574/0 (socket says 172.21.15.173:56574) 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c55f92640 1 -- 172.21.15.173:0/339902229 learned_addr learned my addr 172.21.15.173:0/339902229 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter ms_handle_connect 0x7f0c5009ac40 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter ms_handle_connect 0x7f0c5013b5b0 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter ms_handle_connect 0x7f0c500b1860 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.052+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763472510 0 0) 0x7f0c500ade20 con 0x7f0c5009ac40 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34004eb0 con 0x7f0c5009ac40 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294571401 0 0) 0x7f0c501d7660 con 0x7f0c5013b5b0 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:05.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c500ade20 con 0x7f0c5013b5b0 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162890136 0 0) 0x7f0c501406b0 con 0x7f0c500b1860 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c501d7660 con 0x7f0c500b1860 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3943788666 0 0) 0x7f0c34004eb0 con 0x7f0c5009ac40 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c501406b0 con 0x7f0c5009ac40 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c4c001bc0 con 0x7f0c5009ac40 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4016702983 0 0) 0x7f0c500ade20 con 0x7f0c5013b5b0 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c34004eb0 con 0x7f0c5013b5b0 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c48001a00 con 0x7f0c5013b5b0 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.046 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4005825403 0 0) 0x7f0c501d7660 con 0x7f0c500b1860 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c500ade20 con 0x7f0c500b1860 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c40001870 con 0x7f0c500b1860 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.047 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1459373492 0 0) 0x7f0c501406b0 con 0x7f0c5009ac40 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 >> v1:172.21.15.117:6790/0 conn(0x7f0c500b1860 legacy=0x7f0c5013aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 >> v1:172.21.15.117:6789/0 conn(0x7f0c5013b5b0 legacy=0x7f0c501e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 monclient: found mon.b 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c501cccd0 con 0x7f0c5009ac40 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 10 monclient: _renew_subs 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c501cce60 con 0x7f0c5009ac40 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 10 monclient: _renew_subs 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c501d7190 con 0x7f0c5009ac40 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c581f6640 1 librados: init done 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c4c004bf0 con 0x7f0c5009ac40 2024-03-29T11:19:05.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.053+0000 7f0c47fff640 4 set_mon_vals no callback set 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0c4c004f00 con 0x7f0c5009ac40 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 10 monclient: dump: 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.050 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0c4c0053f0 con 0x7f0c5009ac40 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 949797643 0 0) 0x7f0c4c006770 con 0x7f0c5009ac40 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 10 client.4126.objecter ms_dispatch 0x7f0c50000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c581f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c581f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c581f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c467fc640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.054+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38005080 con 0x7f0c5009ac40 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.056+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0c4c006980 con 0x7f0c5009ac40 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.056+0000 7f0c467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.056+0000 7f0c467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.128+0000 7f0c581f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bfb91a80-d14b-490a-9f7d-4d1e06a97587", "id": 1}] 2024-03-29T11:19:05.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.128+0000 7f0c581f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bfb91a80-d14b-490a-9f7d-4d1e06a97587", "id": 1}] 2024-03-29T11:19:05.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.128+0000 7f0c581f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.128+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bfb91a80-d14b-490a-9f7d-4d1e06a97587", "id": 1} v 0) v1 -- 0x7f0c380058b0 con 0x7f0c5009ac40 2024-03-29T11:19:05.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.136+0000 7f0c467fc640 1 -- 172.21.15.173:0/339902229 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfb91a80-d14b-490a-9f7d-4d1e06a97587", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4053686650 0 940483276) 0x7f0c4c0058b0 con 0x7f0c5009ac40 2024-03-29T11:19:05.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.136+0000 7f0c467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bfb91a80-d14b-490a-9f7d-4d1e06a97587", "id": 1}] 2024-03-29T11:19:05.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.136+0000 7f0c467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.113 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 10 librados: watch_flush enter 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 10 librados: watch_flush exit 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 10 monclient: shutdown 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 >> v1:172.21.15.173:6789/0 conn(0x7f0c5009ac40 legacy=0x7f0c5013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 shutdown_connections 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 >> 172.21.15.173:0/339902229 conn(0x7f0c50066b00 msgr2=0x7f0c5013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 shutdown_connections 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 -- 172.21.15.173:0/339902229 wait complete. 2024-03-29T11:19:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.137+0000 7f0c581f6640 1 librados: shutdown 2024-03-29T11:19:05.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new ea86a613-bb1e-45a8-b681-6a589fa7dc6a 2 2024-03-29T11:19:05.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.247+0000 7f35f7dc1640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.247+0000 7f35f7dc1640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.247+0000 7f35f7dc1640 10 monclient: monmap: 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:05.249067+0000 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:05.249067+0000 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.248+0000 7f35f7dc1640 1 Processor -- start 2024-03-29T11:19:05.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.248+0000 7f35f7dc1640 1 -- start start 2024-03-29T11:19:05.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.248+0000 7f35f7dc1640 10 monclient: init 2024-03-29T11:19:05.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.248+0000 7f35f7dc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.248+0000 7f35f7dc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-a con 0x7f35f009ac40 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-b con 0x7f35f009c870 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-c con 0x7f35f00b4360 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009dec0 con 0x7f35f009ac40 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f009da30 con 0x7f35f00b4360 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b39c0 con 0x7f35f009c870 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient(hunting): authenticate will time out at 814.394684s 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f635e640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f35f00b4360 0x7f35f014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56590/0 (socket says 172.21.15.173:56590) 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f635e640 1 -- 172.21.15.173:0/3909498161 learned_addr learned my addr 172.21.15.173:0/3909498161 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682177860 0 0) 0x7f35f009da30 con 0x7f35f00b4360 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc003520 con 0x7f35f00b4360 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6459772 0 0) 0x7f35f00b39c0 con 0x7f35f009c870 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f009da30 con 0x7f35f009c870 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366922817 0 0) 0x7f35ec001040 con 0x7f35f009ac40 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f00b39c0 con 0x7f35f009ac40 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 110305678 0 0) 0x7f35dc003520 con 0x7f35f00b4360 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 >> v1:172.21.15.117:6790/0 conn(0x7f35f009c870 legacy=0x7f35f013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 >> v1:172.21.15.117:6789/0 conn(0x7f35f009ac40 legacy=0x7f35f00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 monclient: found mon.noname-c 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00b5160 con 0x7f35f00b4360 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f35e8001a10 con 0x7f35f00b4360 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: dump: 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.233 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e8001bf0 con 0x7f35f00b4360 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 1 -- 172.21.15.173:0/3909498161 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f35e8002000 con 0x7f35f00b4360 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f7dc1640 4 set_mon_vals no callback set 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f35f4b5b640 10 monclient: dump: 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.235 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.235 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.235 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.235 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.235 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.249+0000 7f16bc410040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 10 monclient: shutdown 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 1 -- 172.21.15.173:0/3909498161 >> v1:172.21.15.173:6789/0 conn(0x7f35f00b4360 legacy=0x7f35f014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 1 -- 172.21.15.173:0/3909498161 shutdown_connections 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 1 -- 172.21.15.173:0/3909498161 >> 172.21.15.173:0/3909498161 conn(0x7f35f0066b00 msgr2=0x7f35f0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 1 -- 172.21.15.173:0/3909498161 shutdown_connections 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.250+0000 7f35f7dc1640 1 -- 172.21.15.173:0/3909498161 wait complete. 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 10 monclient: monmap: 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 librados: starting msgr at 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 librados: starting objecter 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 Processor -- start 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 -- start start 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 librados: setting wanted keys 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 1 librados: calling monclient init 2024-03-29T11:19:05.242 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.251+0000 7f35f7dc1640 10 monclient: init 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-b con 0x7f35f009ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-c con 0x7f35f00aee50 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient(hunting): picked mon.noname-a con 0x7f35f00ae810 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f01d7b20 con 0x7f35f00ae810 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b00d0 con 0x7f35f009ac40 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00ada70 con 0x7f35f00aee50 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient(hunting): authenticate will time out at 814.397721s 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f5b5d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f35f009ac40 0x7f35f01d7fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56598/0 (socket says 172.21.15.173:56598) 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f5b5d640 1 -- 172.21.15.173:0/492235577 learned_addr learned my addr 172.21.15.173:0/492235577 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter ms_handle_connect 0x7f35f009ac40 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061198921 0 0) 0x7f35f00b00d0 con 0x7f35f009ac40 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.243 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d4004eb0 con 0x7f35f009ac40 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter ms_handle_connect 0x7f35f00aee50 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter ms_handle_connect 0x7f35f00ae810 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75849888 0 0) 0x7f35f00ada70 con 0x7f35f00aee50 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f00b00d0 con 0x7f35f00aee50 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045940103 0 0) 0x7f35f01d7b20 con 0x7f35f00ae810 2024-03-29T11:19:05.244 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f00ada70 con 0x7f35f00ae810 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164508525 0 0) 0x7f35d4004eb0 con 0x7f35f009ac40 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35f01d7b20 con 0x7f35f009ac40 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f35ec001810 con 0x7f35f009ac40 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065512972 0 0) 0x7f35f00b00d0 con 0x7f35f00aee50 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35d4004eb0 con 0x7f35f00aee50 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f35e0001930 con 0x7f35f00aee50 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.246 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3048796541 0 0) 0x7f35f01d7b20 con 0x7f35f009ac40 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 >> v1:172.21.15.117:6790/0 conn(0x7f35f00aee50 legacy=0x7f35f00b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 >> v1:172.21.15.117:6789/0 conn(0x7f35f00ae810 legacy=0x7f35f00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 monclient: found mon.b 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f01d7640 con 0x7f35f009ac40 2024-03-29T11:19:05.247 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _renew_subs 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00b17b0 con 0x7f35f009ac40 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _renew_subs 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f01e3e90 con 0x7f35f009ac40 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35f7dc1640 1 librados: init done 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35ec003610 con 0x7f35f009ac40 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f35ec001d80 con 0x7f35f009ac40 2024-03-29T11:19:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e7fff640 4 set_mon_vals no callback set 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 10 monclient: dump: 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.252+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f35ec003e30 con 0x7f35f009ac40 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2270369922 0 0) 0x7f35ec0057c0 con 0x7f35f009ac40 2024-03-29T11:19:05.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35e67fc640 10 client.4132.objecter ms_dispatch 0x7f35f0000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35e67fc640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35e67fc640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35e67fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35f7dc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35f7dc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35f7dc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.253+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d8005080 con 0x7f35f009ac40 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.255+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f35ec005a20 con 0x7f35f009ac40 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.255+0000 7f35e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.255+0000 7f35e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.250 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.271+0000 7f7664248040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:05.283 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.306+0000 7f16bc410040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:05.301 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.324+0000 7f7664248040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:05.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.328+0000 7f35f7dc1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ea86a613-bb1e-45a8-b681-6a589fa7dc6a", "id": 2}] 2024-03-29T11:19:05.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.328+0000 7f35f7dc1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ea86a613-bb1e-45a8-b681-6a589fa7dc6a", "id": 2}] 2024-03-29T11:19:05.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.328+0000 7f35f7dc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.328+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ea86a613-bb1e-45a8-b681-6a589fa7dc6a", "id": 2} v 0) v1 -- 0x7f35d80058b0 con 0x7f35f009ac40 2024-03-29T11:19:05.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.335+0000 7f35e67fc640 1 -- 172.21.15.173:0/492235577 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea86a613-bb1e-45a8-b681-6a589fa7dc6a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4240139910 0 216599125) 0x7f35ec005c80 con 0x7f35f009ac40 2024-03-29T11:19:05.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.335+0000 7f35e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ea86a613-bb1e-45a8-b681-6a589fa7dc6a", "id": 2}] 2024-03-29T11:19:05.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.335+0000 7f35e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.313 INFO:teuthology.orchestra.run.smithi173.stdout:2 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 10 librados: watch_flush enter 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 10 librados: watch_flush exit 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 10 monclient: shutdown 2024-03-29T11:19:05.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 >> v1:172.21.15.173:6789/0 conn(0x7f35f009ac40 legacy=0x7f35f01d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 shutdown_connections 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 >> 172.21.15.173:0/492235577 conn(0x7f35f0066b00 msgr2=0x7f35f013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 shutdown_connections 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 -- 172.21.15.173:0/492235577 wait complete. 2024-03-29T11:19:05.315 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.336+0000 7f35f7dc1640 1 librados: shutdown 2024-03-29T11:19:05.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new b25cbe47-f48e-4b45-9abb-fa5ebf3e9519 3 2024-03-29T11:19:05.331 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.354+0000 7f16bc410040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:05.346 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.369+0000 7f7664248040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:05.388 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.411+0000 7f16bc410040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:05.399 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.423+0000 7f7664248040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.446+0000 7f83fa8d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.446+0000 7f83fa8d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.446+0000 7f83fa8d6640 10 monclient: monmap: 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:05.448289+0000 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:05.448289+0000 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.425 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.425 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:05.425 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:05.425 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.433 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.447+0000 7f83fa8d6640 1 Processor -- start 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.447+0000 7f83fa8d6640 1 -- start start 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.447+0000 7f83fa8d6640 10 monclient: init 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-b con 0x7f83f409ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-a con 0x7f83f409c870 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-c con 0x7f83f40b4360 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f409dec0 con 0x7f83f409c870 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f409da30 con 0x7f83f40b4360 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f40b39c0 con 0x7f83f409ac40 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient(hunting): authenticate will time out at 814.593944s 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f8e73640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f83f40b4360 0x7f83f414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56604/0 (socket says 172.21.15.173:56604) 2024-03-29T11:19:05.434 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f8e73640 1 -- 172.21.15.173:0/2965228141 learned_addr learned my addr 172.21.15.173:0/2965228141 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288403185 0 0) 0x7f83f409da30 con 0x7f83f40b4360 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83dc003520 con 0x7f83f40b4360 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963311695 0 0) 0x7f83e4001110 con 0x7f83f409c870 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f409da30 con 0x7f83f409c870 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299993494 0 0) 0x7f83f40b39c0 con 0x7f83f409ac40 2024-03-29T11:19:05.435 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e4001110 con 0x7f83f409ac40 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1171894010 0 0) 0x7f83dc003520 con 0x7f83f40b4360 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 >> v1:172.21.15.117:6790/0 conn(0x7f83f409ac40 legacy=0x7f83f40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 >> v1:172.21.15.117:6789/0 conn(0x7f83f409c870 legacy=0x7f83f413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f40b5160 con 0x7f83f40b4360 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.436 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83ec001a10 con 0x7f83f40b4360 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: dump: 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.437 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83ec001bf0 con 0x7f83f40b4360 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 1 -- 172.21.15.173:0/2965228141 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83ec002000 con 0x7f83f40b4360 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83fa8d6640 4 set_mon_vals no callback set 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.448+0000 7f83f2ffd640 10 monclient: dump: 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 10 monclient: shutdown 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.439 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 1 -- 172.21.15.173:0/2965228141 >> v1:172.21.15.173:6789/0 conn(0x7f83f40b4360 legacy=0x7f83f414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 1 -- 172.21.15.173:0/2965228141 shutdown_connections 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 1 -- 172.21.15.173:0/2965228141 >> 172.21.15.173:0/2965228141 conn(0x7f83f4066b00 msgr2=0x7f83f4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 1 -- 172.21.15.173:0/2965228141 shutdown_connections 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.449+0000 7f83fa8d6640 1 -- 172.21.15.173:0/2965228141 wait complete. 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 10 monclient: monmap: 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:05.440 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 librados: starting msgr at 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 librados: starting objecter 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 Processor -- start 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 -- start start 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 librados: setting wanted keys 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 1 librados: calling monclient init 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.450+0000 7f83fa8d6640 10 monclient: init 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-a con 0x7f83f409ac40 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-b con 0x7f83f41d7cd0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient(hunting): picked mon.noname-c con 0x7f83f40ad1e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f41d7130 con 0x7f83f409ac40 2024-03-29T11:19:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f4139b10 con 0x7f83f41d7cd0 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f413ffa0 con 0x7f83f40ad1e0 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83fa8d6640 10 monclient(hunting): authenticate will time out at 814.596984s 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f37fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f83f41d7cd0 0x7f83f4138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56618/0 (socket says 172.21.15.173:56618) 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f37fe640 1 -- 172.21.15.173:0/3425812918 learned_addr learned my addr 172.21.15.173:0/3425812918 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter ms_handle_connect 0x7f83f41d7cd0 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter ms_handle_connect 0x7f83f409ac40 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter ms_handle_connect 0x7f83f40ad1e0 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495860071 0 0) 0x7f83f4139b10 con 0x7f83f41d7cd0 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e0004eb0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855119876 0 0) 0x7f83f41d7130 con 0x7f83f409ac40 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f4139b10 con 0x7f83f409ac40 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062802583 0 0) 0x7f83f413ffa0 con 0x7f83f40ad1e0 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f41d7130 con 0x7f83f40ad1e0 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1428076049 0 0) 0x7f83e0004eb0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83f413ffa0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83e40019a0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.451+0000 7f83f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.444 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1042333730 0 0) 0x7f83f4139b10 con 0x7f83f409ac40 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83e0004eb0 con 0x7f83f409ac40 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761919411 0 0) 0x7f83f41d7130 con 0x7f83f40ad1e0 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83f4139b10 con 0x7f83f40ad1e0 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83e8001830 con 0x7f83f409ac40 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.445 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83ec001d60 con 0x7f83f40ad1e0 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1626848660 0 0) 0x7f83f413ffa0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 >> v1:172.21.15.117:6790/0 conn(0x7f83f40ad1e0 legacy=0x7f83f4139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 >> v1:172.21.15.117:6789/0 conn(0x7f83f409ac40 legacy=0x7f83f41d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 monclient: found mon.b 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f40ae520 con 0x7f83f41d7cd0 2024-03-29T11:19:05.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 10 monclient: _renew_subs 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f40ae800 con 0x7f83f41d7cd0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 10 monclient: _renew_subs 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f4142b70 con 0x7f83f41d7cd0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83fa8d6640 1 librados: init done 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83e4001c40 con 0x7f83f41d7cd0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83e4003a60 con 0x7f83f41d7cd0 2024-03-29T11:19:05.448 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f27fc640 4 set_mon_vals no callback set 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 monclient: dump: 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f83e4004010 con 0x7f83f41d7cd0 2024-03-29T11:19:05.449 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1116682166 0 0) 0x7f83e40067d0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 10 client.4138.objecter ms_dispatch 0x7f83f4000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.452+0000 7f83f0ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.453+0000 7f83fa8d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.453+0000 7f83fa8d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.453+0000 7f83fa8d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.453+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d4005080 con 0x7f83f41d7cd0 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.455+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f83e4006340 con 0x7f83f41d7cd0 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.455+0000 7f83f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.455+0000 7f83f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.451 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.457+0000 7f16bc410040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:05.451 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.465+0000 7f7664248040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.527+0000 7f83fa8d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b25cbe47-f48e-4b45-9abb-fa5ebf3e9519", "id": 3}] 2024-03-29T11:19:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.527+0000 7f83fa8d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b25cbe47-f48e-4b45-9abb-fa5ebf3e9519", "id": 3}] 2024-03-29T11:19:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.527+0000 7f83fa8d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.505 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.527+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b25cbe47-f48e-4b45-9abb-fa5ebf3e9519", "id": 3} v 0) v1 -- 0x7f83d40058b0 con 0x7f83f41d7cd0 2024-03-29T11:19:05.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.535+0000 7f83f0ff9640 1 -- 172.21.15.173:0/3425812918 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b25cbe47-f48e-4b45-9abb-fa5ebf3e9519", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1462013624 0 525046306) 0x7f83e4006620 con 0x7f83f41d7cd0 2024-03-29T11:19:05.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.535+0000 7f83f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b25cbe47-f48e-4b45-9abb-fa5ebf3e9519", "id": 3}] 2024-03-29T11:19:05.512 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.535+0000 7f83f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.512 INFO:teuthology.orchestra.run.smithi173.stdout:3 2024-03-29T11:19:05.513 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 10 librados: watch_flush enter 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 10 librados: watch_flush exit 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 10 monclient: shutdown 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 >> v1:172.21.15.173:6789/0 conn(0x7f83f41d7cd0 legacy=0x7f83f4138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 shutdown_connections 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 >> 172.21.15.173:0/3425812918 conn(0x7f83f4066b00 msgr2=0x7f83f40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 shutdown_connections 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 -- 172.21.15.173:0/3425812918 wait complete. 2024-03-29T11:19:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.536+0000 7f83fa8d6640 1 librados: shutdown 2024-03-29T11:19:05.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 582ed39e-64ed-42fb-90cf-5936b771756c 4 2024-03-29T11:19:05.555 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.577+0000 7f16bc410040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:05.557 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.581+0000 7f7664248040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:05.626 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.649+0000 7f7664248040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:05.628 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.651+0000 7f16bc410040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 10 monclient: monmap: 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:05.678578+0000 2024-03-29T11:19:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:05.678578+0000 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:05.655 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 1 Processor -- start 2024-03-29T11:19:05.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 1 -- start start 2024-03-29T11:19:05.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.677+0000 7fdef31d6640 10 monclient: init 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-a con 0x7fdeec09ac40 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-b con 0x7fdeec09c870 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-c con 0x7fdeec0b4360 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec09dec0 con 0x7fdeec09ac40 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec09da30 con 0x7fdeec0b4360 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec0b39c0 con 0x7fdeec09c870 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef31d6640 10 monclient(hunting): authenticate will time out at 814.824160s 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef1773640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fdeec0b4360 0x7fdeec14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56634/0 (socket says 172.21.15.173:56634) 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdef1773640 1 -- 172.21.15.173:0/1228848541 learned_addr learned my addr 172.21.15.173:0/1228848541 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712326100 0 0) 0x7fdeec09da30 con 0x7fdeec0b4360 2024-03-29T11:19:05.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.678+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdedc003520 con 0x7fdeec0b4360 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036416632 0 0) 0x7fdee0001110 con 0x7fdeec09ac40 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec09da30 con 0x7fdeec09ac40 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419430468 0 0) 0x7fdeec0b39c0 con 0x7fdeec09c870 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0001110 con 0x7fdeec09c870 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280322892 0 0) 0x7fdedc003520 con 0x7fdeec0b4360 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 >> v1:172.21.15.117:6790/0 conn(0x7fdeec09c870 legacy=0x7fdeec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 >> v1:172.21.15.117:6789/0 conn(0x7fdeec09ac40 legacy=0x7fdeec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0b5160 con 0x7fdeec0b4360 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdee4001a10 con 0x7fdeec0b4360 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: dump: 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.667 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdee4001bf0 con 0x7fdeec0b4360 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 1 -- 172.21.15.173:0/1228848541 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdee4002000 con 0x7fdeec0b4360 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 4 set_mon_vals no callback set 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdeeb7fe640 10 monclient: dump: 2024-03-29T11:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 10 monclient: shutdown 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 1 -- 172.21.15.173:0/1228848541 >> v1:172.21.15.173:6789/0 conn(0x7fdeec0b4360 legacy=0x7fdeec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 1 -- 172.21.15.173:0/1228848541 shutdown_connections 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.679+0000 7fdef31d6640 1 -- 172.21.15.173:0/1228848541 >> 172.21.15.173:0/1228848541 conn(0x7fdeec066b00 msgr2=0x7fdeec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.680+0000 7fdef31d6640 1 -- 172.21.15.173:0/1228848541 shutdown_connections 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.680+0000 7fdef31d6640 1 -- 172.21.15.173:0/1228848541 wait complete. 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.680+0000 7fdef31d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.680+0000 7fdef31d6640 10 monclient: monmap: 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 librados: starting msgr at 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 librados: starting objecter 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 Processor -- start 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 -- start start 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 librados: setting wanted keys 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 1 librados: calling monclient init 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.681+0000 7fdef31d6640 10 monclient: init 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-b con 0x7fdeec09ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-a con 0x7fdeec141910 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient(hunting): picked mon.noname-c con 0x7fdeec13ffa0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec139b60 con 0x7fdeec141910 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec1d6f10 con 0x7fdeec09ac40 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec0afba0 con 0x7fdeec13ffa0 2024-03-29T11:19:05.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef31d6640 10 monclient(hunting): authenticate will time out at 814.828193s 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef0f72640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fdeec09ac40 0x7fdeec141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56642/0 (socket says 172.21.15.173:56642) 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdef0f72640 1 -- 172.21.15.173:0/2662860085 learned_addr learned my addr 172.21.15.173:0/2662860085 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter ms_handle_connect 0x7fdeec09ac40 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter ms_handle_connect 0x7fdeec13ffa0 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter ms_handle_connect 0x7fdeec141910 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888662570 0 0) 0x7fdeec1d6f10 con 0x7fdeec09ac40 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded0004eb0 con 0x7fdeec09ac40 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058970865 0 0) 0x7fdeec0afba0 con 0x7fdeec13ffa0 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec1d6f10 con 0x7fdeec13ffa0 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.682+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303345126 0 0) 0x7fdee0001110 con 0x7fdeec141910 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec0afba0 con 0x7fdeec141910 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2874383930 0 0) 0x7fded0004eb0 con 0x7fdeec09ac40 2024-03-29T11:19:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdee0001110 con 0x7fdeec09ac40 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdee0001a20 con 0x7fdeec09ac40 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616943710 0 0) 0x7fdeec1d6f10 con 0x7fdeec13ffa0 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded0004eb0 con 0x7fdeec13ffa0 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdee4001f00 con 0x7fdeec13ffa0 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.675 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219325102 0 0) 0x7fdeec0afba0 con 0x7fdeec141910 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdeec1d6f10 con 0x7fdeec141910 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fded80015d0 con 0x7fdeec141910 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.676 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1247465427 0 0) 0x7fdee0001110 con 0x7fdeec09ac40 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 >> v1:172.21.15.117:6790/0 conn(0x7fdeec13ffa0 legacy=0x7fdeec138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 >> v1:172.21.15.117:6789/0 conn(0x7fdeec141910 legacy=0x7fdeec1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 monclient: found mon.b 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec139520 con 0x7fdeec09ac40 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 10 monclient: _renew_subs 2024-03-29T11:19:05.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec139710 con 0x7fdeec09ac40 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 10 monclient: _renew_subs 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec1d7c90 con 0x7fdeec09ac40 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdef31d6640 1 librados: init done 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdee0001ce0 con 0x7fdeec09ac40 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdee0003a60 con 0x7fdeec09ac40 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdeeaffd640 4 set_mon_vals no callback set 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.683+0000 7fdee97fa640 10 monclient: dump: 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.678 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdee0004010 con 0x7fdeec09ac40 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2487497892 0 0) 0x7fdee00068f0 con 0x7fdeec09ac40 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 10 client.4144.objecter ms_dispatch 0x7fdeec000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdee97fa640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdef31d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdef31d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdef31d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.684+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdecc005080 con 0x7fdeec09ac40 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.686+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdee0006bd0 con 0x7fdeec09ac40 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.686+0000 7fdee97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.686+0000 7fdee97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.680 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.701+0000 7f16bc410040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:05.681 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.695+0000 7f7664248040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:05.724 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.747+0000 7f7664248040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:05.731 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.754+0000 7f16bc410040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.757+0000 7fdef31d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "582ed39e-64ed-42fb-90cf-5936b771756c", "id": 4}] 2024-03-29T11:19:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.757+0000 7fdef31d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "582ed39e-64ed-42fb-90cf-5936b771756c", "id": 4}] 2024-03-29T11:19:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.757+0000 7fdef31d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.757+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "582ed39e-64ed-42fb-90cf-5936b771756c", "id": 4} v 0) v1 -- 0x7fdecc0058b0 con 0x7fdeec09ac40 2024-03-29T11:19:05.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.764+0000 7fdee97fa640 1 -- 172.21.15.173:0/2662860085 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "582ed39e-64ed-42fb-90cf-5936b771756c", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3019991683 0 1697012583) 0x7fdee00064b0 con 0x7fdeec09ac40 2024-03-29T11:19:05.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.764+0000 7fdee97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "582ed39e-64ed-42fb-90cf-5936b771756c", "id": 4}] 2024-03-29T11:19:05.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.764+0000 7fdee97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.742 INFO:teuthology.orchestra.run.smithi173.stdout:4 2024-03-29T11:19:05.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 10 librados: watch_flush enter 2024-03-29T11:19:05.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 10 librados: watch_flush exit 2024-03-29T11:19:05.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.743 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 10 monclient: shutdown 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.765+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 >> v1:172.21.15.173:6789/0 conn(0x7fdeec09ac40 legacy=0x7fdeec141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.766+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 shutdown_connections 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.766+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 >> 172.21.15.173:0/2662860085 conn(0x7fdeec066b00 msgr2=0x7fdeec13c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.766+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 shutdown_connections 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.766+0000 7fdef31d6640 1 -- 172.21.15.173:0/2662860085 wait complete. 2024-03-29T11:19:05.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.766+0000 7fdef31d6640 1 librados: shutdown 2024-03-29T11:19:05.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new bbe2fc3d-7406-4ebd-8354-d20d160ef309 5 2024-03-29T11:19:05.772 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:05.795+0000 7f7664248040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:05.782 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:05.805+0000 7f16bc410040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.876+0000 7f4757be8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.876+0000 7f4757be8640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.876+0000 7f4757be8640 10 monclient: monmap: 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:05.877674+0000 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:05.877674+0000 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:05.854 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.876+0000 7f4757be8640 1 Processor -- start 2024-03-29T11:19:05.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 1 -- start start 2024-03-29T11:19:05.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient: init 2024-03-29T11:19:05.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:05.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-b con 0x7f475009ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-a con 0x7f475009c870 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-c con 0x7f47500b4360 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475009dec0 con 0x7f475009c870 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475009da30 con 0x7f47500b4360 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500b39c0 con 0x7f475009ac40 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4757be8640 10 monclient(hunting): authenticate will time out at 815.023283s 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4756185640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f47500b4360 0x7f475014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56650/0 (socket says 172.21.15.173:56650) 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.877+0000 7f4756185640 1 -- 172.21.15.173:0/1587537848 learned_addr learned my addr 172.21.15.173:0/1587537848 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031564784 0 0) 0x7f475009da30 con 0x7f47500b4360 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473c003520 con 0x7f47500b4360 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842586625 0 0) 0x7f47500b39c0 con 0x7f475009ac40 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f475009da30 con 0x7f475009ac40 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589134554 0 0) 0x7f4740001110 con 0x7f475009c870 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500b39c0 con 0x7f475009c870 2024-03-29T11:19:05.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705398983 0 0) 0x7f473c003520 con 0x7f47500b4360 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 >> v1:172.21.15.117:6790/0 conn(0x7f475009ac40 legacy=0x7f47500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 >> v1:172.21.15.117:6789/0 conn(0x7f475009c870 legacy=0x7f475013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 monclient: found mon.noname-c 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47500b5160 con 0x7f47500b4360 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4748001a10 con 0x7f47500b4360 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4757be8640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4757be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: dump: 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4757be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4748001bf0 con 0x7f47500b4360 2024-03-29T11:19:05.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 1 -- 172.21.15.173:0/1587537848 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4748002000 con 0x7f47500b4360 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4757be8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4757be8640 4 set_mon_vals no callback set 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.878+0000 7f4754982640 10 monclient: dump: 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.868 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 10 monclient: shutdown 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 1 -- 172.21.15.173:0/1587537848 >> v1:172.21.15.173:6789/0 conn(0x7f47500b4360 legacy=0x7f475014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 1 -- 172.21.15.173:0/1587537848 shutdown_connections 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 1 -- 172.21.15.173:0/1587537848 >> 172.21.15.173:0/1587537848 conn(0x7f4750066b00 msgr2=0x7f4750066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 1 -- 172.21.15.173:0/1587537848 shutdown_connections 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 1 -- 172.21.15.173:0/1587537848 wait complete. 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 10 monclient: build_initial_monmap 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.879+0000 7f4757be8640 10 monclient: monmap: 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 librados: starting msgr at 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 librados: starting objecter 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 Processor -- start 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 -- start start 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 librados: setting wanted keys 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 librados: calling monclient init 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient: init 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:05.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-b con 0x7f475009ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-c con 0x7f47501d8690 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient(hunting): picked mon.noname-a con 0x7f47501455c0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47501d7af0 con 0x7f47501455c0 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4750145a10 con 0x7f475009ac40 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500b5a20 con 0x7f47501d8690 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4757be8640 10 monclient(hunting): authenticate will time out at 815.026342s 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4755984640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f475009ac40 0x7f47501d7f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56664/0 (socket says 172.21.15.173:56664) 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f4755984640 1 -- 172.21.15.173:0/1864088223 learned_addr learned my addr 172.21.15.173:0/1864088223 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter ms_handle_connect 0x7f475009ac40 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter ms_handle_connect 0x7f47501455c0 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.871 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter ms_handle_connect 0x7f47501d8690 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.880+0000 7f47467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374340721 0 0) 0x7f4750145a10 con 0x7f475009ac40 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734004eb0 con 0x7f475009ac40 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731121883 0 0) 0x7f47501d7af0 con 0x7f47501455c0 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4750145a10 con 0x7f47501455c0 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485062423 0 0) 0x7f47500b5a20 con 0x7f47501d8690 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:05.872 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47501d7af0 con 0x7f47501d8690 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3881419526 0 0) 0x7f4734004eb0 con 0x7f475009ac40 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47500b5a20 con 0x7f475009ac40 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f474c0019b0 con 0x7f475009ac40 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.873 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079378980 0 0) 0x7f47501d7af0 con 0x7f47501d8690 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4734004eb0 con 0x7f47501d8690 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095602648 0 0) 0x7f4750145a10 con 0x7f47501455c0 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47501d7af0 con 0x7f47501455c0 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4740001860 con 0x7f47501d8690 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.874 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4748001f00 con 0x7f47501455c0 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient(hunting): dump: 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.875 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 590176974 0 0) 0x7f47500b5a20 con 0x7f475009ac40 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 >> v1:172.21.15.117:6790/0 conn(0x7f47501d8690 legacy=0x7f4750144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 >> v1:172.21.15.117:6789/0 conn(0x7f47501455c0 legacy=0x7f47501e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 monclient: found mon.b 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.876 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750141960 con 0x7f475009ac40 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 10 monclient: _renew_subs 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47501e3b20 con 0x7f475009ac40 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 10 monclient: _renew_subs 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47501e3e90 con 0x7f475009ac40 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4757be8640 1 librados: init done 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f474c003610 con 0x7f475009ac40 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f474c003920 con 0x7f475009ac40 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f4747fff640 4 set_mon_vals no callback set 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 monclient: dump: 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:05.878 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f474c003ed0 con 0x7f475009ac40 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2000567853 0 0) 0x7f474c006710 con 0x7f475009ac40 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 10 client.4150.objecter ms_dispatch 0x7f4750000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.881+0000 7f47467fc640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.882+0000 7f4757be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.882+0000 7f4757be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.882+0000 7f4757be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.882+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4738005080 con 0x7f475009ac40 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.884+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f474c0042c0 con 0x7f475009ac40 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.884+0000 7f47467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:05.879 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.884+0000 7f47467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:05.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.956+0000 7f4757be8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bbe2fc3d-7406-4ebd-8354-d20d160ef309", "id": 5}] 2024-03-29T11:19:05.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.956+0000 7f4757be8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bbe2fc3d-7406-4ebd-8354-d20d160ef309", "id": 5}] 2024-03-29T11:19:05.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.956+0000 7f4757be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:05.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.956+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bbe2fc3d-7406-4ebd-8354-d20d160ef309", "id": 5} v 0) v1 -- 0x7f47380058b0 con 0x7f475009ac40 2024-03-29T11:19:05.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.964+0000 7f47467fc640 1 -- 172.21.15.173:0/1864088223 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bbe2fc3d-7406-4ebd-8354-d20d160ef309", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2366198272 0 1988412176) 0x7f474c006190 con 0x7f475009ac40 2024-03-29T11:19:05.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.964+0000 7f47467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bbe2fc3d-7406-4ebd-8354-d20d160ef309", "id": 5}] 2024-03-29T11:19:05.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.964+0000 7f47467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:05.941 INFO:teuthology.orchestra.run.smithi173.stdout:5 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 10 librados: watch_flush enter 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 10 librados: watch_flush exit 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 10 monclient: shutdown 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 >> v1:172.21.15.173:6789/0 conn(0x7f475009ac40 legacy=0x7f47501d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 shutdown_connections 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 >> 172.21.15.173:0/1864088223 conn(0x7f4750066b00 msgr2=0x7f475013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:05.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 shutdown_connections 2024-03-29T11:19:05.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 -- 172.21.15.173:0/1864088223 wait complete. 2024-03-29T11:19:05.944 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:05.965+0000 7f4757be8640 1 librados: shutdown 2024-03-29T11:19:05.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 12944eb6-17fc-42fa-a782-e5dbd4783762 6 2024-03-29T11:19:06.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 10 monclient: monmap: 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:06.117546+0000 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:06.117546+0000 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 1 Processor -- start 2024-03-29T11:19:06.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 1 -- start start 2024-03-29T11:19:06.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.116+0000 7fcee06e4640 10 monclient: init 2024-03-29T11:19:06.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:06.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-b con 0x7fced809ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.103 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-c con 0x7fced809c870 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-a con 0x7fced80b4360 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced809dec0 con 0x7fced80b4360 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced809da30 con 0x7fced809c870 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80b39c0 con 0x7fced809ac40 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcee06e4640 10 monclient(hunting): authenticate will time out at 815.263171s 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fceddc7f640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fced809c870 0x7fced813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56680/0 (socket says 172.21.15.173:56680) 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fceddc7f640 1 -- 172.21.15.173:0/2041421543 learned_addr learned my addr 172.21.15.173:0/2041421543 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945750234 0 0) 0x7fced809da30 con 0x7fced809c870 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:19:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.117+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec4003520 con 0x7fced809c870 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852604089 0 0) 0x7fced0001490 con 0x7fced80b4360 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced809da30 con 0x7fced80b4360 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343016813 0 0) 0x7fced80b39c0 con 0x7fced809ac40 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced0001490 con 0x7fced809ac40 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1251927670 0 0) 0x7fcec4003520 con 0x7fced809c870 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 >> v1:172.21.15.117:6790/0 conn(0x7fced809ac40 legacy=0x7fced80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 >> v1:172.21.15.117:6789/0 conn(0x7fced80b4360 legacy=0x7fced814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.105 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 monclient: found mon.noname-c 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80b5160 con 0x7fced809c870 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fcec8001690 con 0x7fced809c870 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: dump: 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.106 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec8001870 con 0x7fced809c870 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 4 set_mon_vals no callback set 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 1 -- 172.21.15.173:0/2041421543 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fcec8001c80 con 0x7fced809c870 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.107 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcedd47e640 10 monclient: dump: 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 10 monclient: shutdown 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 1 -- 172.21.15.173:0/2041421543 >> v1:172.21.15.173:6789/0 conn(0x7fced809c870 legacy=0x7fced813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 1 -- 172.21.15.173:0/2041421543 shutdown_connections 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.118+0000 7fcee06e4640 1 -- 172.21.15.173:0/2041421543 >> 172.21.15.173:0/2041421543 conn(0x7fced8066b00 msgr2=0x7fced8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 1 -- 172.21.15.173:0/2041421543 shutdown_connections 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 1 -- 172.21.15.173:0/2041421543 wait complete. 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 10 monclient: monmap: 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.109 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 1 librados: starting msgr at 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 1 librados: starting objecter 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.119+0000 7fcee06e4640 1 Processor -- start 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 -- start start 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 librados: setting wanted keys 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 librados: calling monclient init 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient: init 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-c con 0x7fced809ac40 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-b con 0x7fced80af4e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient(hunting): picked mon.noname-a con 0x7fced81d7cf0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80b1940 con 0x7fced81d7cf0 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced81d70c0 con 0x7fced80af4e0 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80acd00 con 0x7fced809ac40 2024-03-29T11:19:06.110 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fcee06e4640 10 monclient(hunting): authenticate will time out at 815.266182s 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceddc7f640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fced80af4e0 0x7fced81d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56694/0 (socket says 172.21.15.173:56694) 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceddc7f640 1 -- 172.21.15.173:0/3055307312 learned_addr learned my addr 172.21.15.173:0/3055307312 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced80af4e0 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced809ac40 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced81d7cf0 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337017437 0 0) 0x7fced81d70c0 con 0x7fced80af4e0 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc004eb0 con 0x7fced80af4e0 2024-03-29T11:19:06.111 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699848276 0 0) 0x7fced80b1940 con 0x7fced81d7cf0 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced81d70c0 con 0x7fced81d7cf0 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218848755 0 0) 0x7fced80acd00 con 0x7fced809ac40 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.120+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced80b1940 con 0x7fced809ac40 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3982745154 0 0) 0x7fcebc004eb0 con 0x7fced80af4e0 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced80acd00 con 0x7fced80af4e0 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fcec8001bc0 con 0x7fced80af4e0 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.112 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): dump: 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707602141 0 0) 0x7fced81d70c0 con 0x7fced81d7cf0 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcebc004eb0 con 0x7fced81d7cf0 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fced0001b40 con 0x7fced81d7cf0 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.113 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient(hunting): dump: 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595228349 0 0) 0x7fced80b1940 con 0x7fced809ac40 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced81d70c0 con 0x7fced809ac40 2024-03-29T11:19:06.114 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2021355670 0 0) 0x7fced80acd00 con 0x7fced80af4e0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 >> v1:172.21.15.117:6790/0 conn(0x7fced809ac40 legacy=0x7fced80b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 >> v1:172.21.15.117:6789/0 conn(0x7fced81d7cf0 legacy=0x7fced81e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 monclient: found mon.b 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80af970 con 0x7fced80af4e0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 10 monclient: _renew_subs 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced81e3260 con 0x7fced80af4e0 2024-03-29T11:19:06.115 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 10 monclient: _renew_subs 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec8004bf0 con 0x7fced80af4e0 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fced81e35d0 con 0x7fced80af4e0 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcee06e4640 1 librados: init done 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fcec8004f00 con 0x7fced80af4e0 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 monclient: dump: 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.116 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fcedcc7d640 4 set_mon_vals no callback set 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcec80053f0 con 0x7fced80af4e0 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 186168396 0 0) 0x7fcec8005c90 con 0x7fced80af4e0 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 10 client.4156.objecter ms_dispatch 0x7fced8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.121+0000 7fceceffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.122+0000 7fcee06e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.122+0000 7fcee06e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.122+0000 7fcee06e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.122+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced81d7300 con 0x7fced80af4e0 2024-03-29T11:19:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.124+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fced81d7300 con 0x7fced80af4e0 2024-03-29T11:19:06.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.124+0000 7fceceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.118 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.124+0000 7fceceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:06.156 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.179+0000 7f7664248040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:06.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.196+0000 7fcee06e4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "12944eb6-17fc-42fa-a782-e5dbd4783762", "id": 6}] 2024-03-29T11:19:06.173 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.196+0000 7fcee06e4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "12944eb6-17fc-42fa-a782-e5dbd4783762", "id": 6}] 2024-03-29T11:19:06.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.196+0000 7fcee06e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.174 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.196+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "12944eb6-17fc-42fa-a782-e5dbd4783762", "id": 6} v 0) v1 -- 0x7fced809b020 con 0x7fced80af4e0 2024-03-29T11:19:06.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.200+0000 7fceceffd640 1 -- 172.21.15.173:0/3055307312 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "12944eb6-17fc-42fa-a782-e5dbd4783762", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 985092955 0 1113811849) 0x7fcec8006000 con 0x7fced80af4e0 2024-03-29T11:19:06.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.200+0000 7fceceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "12944eb6-17fc-42fa-a782-e5dbd4783762", "id": 6}] 2024-03-29T11:19:06.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.200+0000 7fceceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:06.177 INFO:teuthology.orchestra.run.smithi173.stdout:6 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 10 librados: watch_flush enter 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 10 librados: watch_flush exit 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 10 monclient: shutdown 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 >> v1:172.21.15.173:6789/0 conn(0x7fced80af4e0 legacy=0x7fced81d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 shutdown_connections 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 >> 172.21.15.173:0/3055307312 conn(0x7fced8066b00 msgr2=0x7fced813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 shutdown_connections 2024-03-29T11:19:06.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 -- 172.21.15.173:0/3055307312 wait complete. 2024-03-29T11:19:06.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.201+0000 7fcee06e4640 1 librados: shutdown 2024-03-29T11:19:06.185 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.207+0000 7f16bc410040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:06.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new b93c3c95-e26b-4ebe-b527-2271096e491d 7 2024-03-29T11:19:06.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.311+0000 7f10f174d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.311+0000 7f10f174d640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.311+0000 7f10f174d640 10 monclient: monmap: 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:19:06.312975+0000 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:19:06.312975+0000 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:06.289 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.295 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.315+0000 7f7664248040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:06.298 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 1 Processor -- start 2024-03-29T11:19:06.298 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 1 -- start start 2024-03-29T11:19:06.298 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient: init 2024-03-29T11:19:06.298 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.298 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec0b8240 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec0b1f60 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec09c8c0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec09bd80 con 0x7f10ec09c8c0 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec09b960 con 0x7f10ec0b1f60 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0b2ec0 con 0x7f10ec0b8240 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.312+0000 7f10f174d640 10 monclient(hunting): authenticate will time out at 815.458531s 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10e3fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f10ec0b1f60 0x7f10ec09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56706/0 (socket says 172.21.15.173:56706) 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10e3fff640 1 -- 172.21.15.173:0/924968306 learned_addr learned my addr 172.21.15.173:0/924968306 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543030479 0 0) 0x7f10ec09b960 con 0x7f10ec0b1f60 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d8003520 con 0x7f10ec0b1f60 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392053069 0 0) 0x7f10ec0b2ec0 con 0x7f10ec0b8240 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec09b960 con 0x7f10ec0b8240 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114438147 0 0) 0x7f10ec09bd80 con 0x7f10ec09c8c0 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec0b2ec0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644527227 0 0) 0x7f10d8003520 con 0x7f10ec0b1f60 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 >> v1:172.21.15.117:6790/0 conn(0x7f10ec0b8240 legacy=0x7f10ec13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 >> v1:172.21.15.117:6789/0 conn(0x7f10ec09c8c0 legacy=0x7f10ec14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec13fb60 con 0x7f10ec0b1f60 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f10d4001690 con 0x7f10ec0b1f60 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10f174d640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10f174d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: dump: 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10f174d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10d4001870 con 0x7f10ec0b1f60 2024-03-29T11:19:06.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10f174d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 1 -- 172.21.15.173:0/924968306 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f10d4001c80 con 0x7f10ec0b1f60 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10f174d640 4 set_mon_vals no callback set 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.313+0000 7f10ea7fc640 10 monclient: dump: 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.303 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 10 monclient: shutdown 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 1 -- 172.21.15.173:0/924968306 >> v1:172.21.15.173:6789/0 conn(0x7f10ec0b1f60 legacy=0x7f10ec09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 1 -- 172.21.15.173:0/924968306 shutdown_connections 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 1 -- 172.21.15.173:0/924968306 >> 172.21.15.173:0/924968306 conn(0x7f10ec066b00 msgr2=0x7f10ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 1 -- 172.21.15.173:0/924968306 shutdown_connections 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 1 -- 172.21.15.173:0/924968306 wait complete. 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.314+0000 7f10f174d640 10 monclient: monmap: 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 librados: starting msgr at 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 librados: starting objecter 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 Processor -- start 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 -- start start 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 librados: setting wanted keys 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 librados: calling monclient init 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient: init 2024-03-29T11:19:06.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec09c8c0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec0b8240 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec1d3e50 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec133890 con 0x7f10ec0b8240 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec0ae240 con 0x7f10ec09c8c0 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec135630 con 0x7f10ec1d3e50 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.315+0000 7f10f174d640 10 monclient(hunting): authenticate will time out at 815.461579s 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10eaffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f10ec09c8c0 0x7f10ec1d7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56720/0 (socket says 172.21.15.173:56720) 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10eaffd640 1 -- 172.21.15.173:0/733778315 learned_addr learned my addr 172.21.15.173:0/733778315 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter ms_handle_connect 0x7f10ec09c8c0 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter ms_handle_connect 0x7f10ec0b8240 2024-03-29T11:19:06.306 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436209156 0 0) 0x7f10ec0ae240 con 0x7f10ec09c8c0 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): my global_id is 4162 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc004eb0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter ms_handle_connect 0x7f10ec1d3e50 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656720547 0 0) 0x7f10ec13fb60 con 0x7f10ec0b8240 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec0ae240 con 0x7f10ec0b8240 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723545748 0 0) 0x7f10ec135630 con 0x7f10ec1d3e50 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec13fb60 con 0x7f10ec1d3e50 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963823239 0 0) 0x7f10cc004eb0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec135630 con 0x7f10ec09c8c0 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f10dc001900 con 0x7f10ec09c8c0 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325787180 0 0) 0x7f10ec0ae240 con 0x7f10ec0b8240 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc004eb0 con 0x7f10ec0b8240 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f10d40019e0 con 0x7f10ec0b8240 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:06.309 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078357889 0 0) 0x7f10ec13fb60 con 0x7f10ec1d3e50 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10ec0ae240 con 0x7f10ec1d3e50 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3516322249 0 0) 0x7f10ec135630 con 0x7f10ec09c8c0 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 >> v1:172.21.15.117:6790/0 conn(0x7f10ec1d3e50 legacy=0x7f10ec1e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.310 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 >> v1:172.21.15.117:6789/0 conn(0x7f10ec0b8240 legacy=0x7f10ec1d4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 monclient: found mon.b 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec1357c0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 5 monclient: authenticate success, global_id 4162 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 10 monclient: _renew_subs 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec1d7f60 con 0x7f10ec09c8c0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 10 monclient: _renew_subs 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.311 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec1d4d80 con 0x7f10ec09c8c0 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10f174d640 1 librados: init done 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10dc003610 con 0x7f10ec09c8c0 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f10dc003960 con 0x7f10ec09c8c0 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 monclient: dump: 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.312 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f10dc003e50 con 0x7f10ec09c8c0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2718839742 0 0) 0x7f10dc0057c0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 10 client.4162.objecter ms_dispatch 0x7f10ec000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.316+0000 7f10e37fe640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.317+0000 7f10e37fe640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.317+0000 7f10f174d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.317+0000 7f10f174d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.317+0000 7f10f174d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.317+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d0005080 con 0x7f10ec09c8c0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.319+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f10dc005980 con 0x7f10ec09c8c0 2024-03-29T11:19:06.313 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.319+0000 7f10e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.319+0000 7f10e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:06.327 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.350+0000 7f16bc410040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:06.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.392+0000 7f10f174d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b93c3c95-e26b-4ebe-b527-2271096e491d", "id": 7}] 2024-03-29T11:19:06.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.392+0000 7f10f174d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b93c3c95-e26b-4ebe-b527-2271096e491d", "id": 7}] 2024-03-29T11:19:06.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.392+0000 7f10f174d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:06.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.392+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b93c3c95-e26b-4ebe-b527-2271096e491d", "id": 7} v 0) v1 -- 0x7f10d00058b0 con 0x7f10ec09c8c0 2024-03-29T11:19:06.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10e37fe640 1 -- 172.21.15.173:0/733778315 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b93c3c95-e26b-4ebe-b527-2271096e491d", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2366037772 0 1371665406) 0x7f10dc004100 con 0x7f10ec09c8c0 2024-03-29T11:19:06.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b93c3c95-e26b-4ebe-b527-2271096e491d", "id": 7}] 2024-03-29T11:19:06.373 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:06.373 INFO:teuthology.orchestra.run.smithi173.stdout:7 2024-03-29T11:19:06.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 10 librados: watch_flush enter 2024-03-29T11:19:06.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 10 librados: watch_flush exit 2024-03-29T11:19:06.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 10 client.4162.objecter successfully canceled tick 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 10 monclient: shutdown 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 >> v1:172.21.15.173:6789/0 conn(0x7f10ec09c8c0 legacy=0x7f10ec1d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 shutdown_connections 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.396+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 >> 172.21.15.173:0/733778315 conn(0x7f10ec066b00 msgr2=0x7f10ec13dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.397+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 shutdown_connections 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.397+0000 7f10f174d640 1 -- 172.21.15.173:0/733778315 wait complete. 2024-03-29T11:19:06.375 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T11:19:06.397+0000 7f10f174d640 1 librados: shutdown 2024-03-29T11:19:06.384 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:06.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:06.386 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:06.387 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:06.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:06.389 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:06.389 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:06.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:06.391 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:06.391 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:06.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:06.394 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:06.394 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:06.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:06.396 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:06.396 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:06.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:06.398 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:06.398 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:06.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:06.400 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:06.401 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:06.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:06.403 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:06.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:06.418 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.442+0000 7f7664248040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:06.452 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.475+0000 7f16bc410040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:06.492 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.515+0000 7f7664248040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:06.505 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.528+0000 7f16bc410040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:06.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.548+0000 7fe9aa97b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:06.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.548+0000 7fe9aa97b640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.525 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.548+0000 7fe9aa97b640 10 monclient: monmap: 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:19:06.549821+0000 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:19:06.549821+0000 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:06.526 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.549+0000 7fe9aa97b640 1 Processor -- start 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.549+0000 7fe9aa97b640 1 -- start start 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.549+0000 7fe9aa97b640 10 monclient: init 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a409ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a409c870 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a37fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe9a409c870 0x7fe9a413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40822/0 (socket says 172.21.15.117:40822) 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a37fe640 1 -- 172.21.15.117:0/1138314482 learned_addr learned my addr 172.21.15.117:0/1138314482 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a40b4360 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a409dec0 con 0x7fe9a40b4360 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a409da30 con 0x7fe9a409ac40 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a40b39c0 con 0x7fe9a409c870 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9aa97b640 10 monclient(hunting): authenticate will time out at 814.859559s 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556283503 0 0) 0x7fe9a40b39c0 con 0x7fe9a409c870 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:19:06.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c003520 con 0x7fe9a409c870 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573228342 0 0) 0x7fe99c001490 con 0x7fe9a40b4360 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a40b39c0 con 0x7fe9a40b4360 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621790430 0 0) 0x7fe9a409da30 con 0x7fe9a409ac40 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c001490 con 0x7fe9a409ac40 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2627781692 0 0) 0x7fe98c003520 con 0x7fe9a409c870 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 >> v1:172.21.15.173:6789/0 conn(0x7fe9a409ac40 legacy=0x7fe9a40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 >> v1:172.21.15.117:6789/0 conn(0x7fe9a40b4360 legacy=0x7fe9a414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.550+0000 7fe9a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40b5160 con 0x7fe9a409c870 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 5 monclient: authenticate success, global_id 4178 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe994001690 con 0x7fe9a409c870 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:06.537 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: dump: 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9940017f0 con 0x7fe9a409c870 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.538 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 1 -- 172.21.15.117:0/1138314482 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe994001c40 con 0x7fe9a409c870 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 4 set_mon_vals no callback set 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9a2ffd640 10 monclient: dump: 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.539 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 10 monclient: shutdown 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 >> v1:172.21.15.117:6790/0 conn(0x7fe9a409c870 legacy=0x7fe9a413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 shutdown_connections 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 >> 172.21.15.117:0/1138314482 conn(0x7fe9a4066b00 msgr2=0x7fe9a4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 shutdown_connections 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.551+0000 7fe9aa97b640 1 -- 172.21.15.117:0/1138314482 wait complete. 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 10 monclient: build_initial_monmap 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 10 monclient: monmap: 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:06.540 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 librados: starting msgr at 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 librados: starting objecter 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 Processor -- start 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 -- start start 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 librados: setting wanted keys 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 1 librados: calling monclient init 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.552+0000 7fe9aa97b640 10 monclient: init 2024-03-29T11:19:06.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a409ac40 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a40b0fb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a41d74b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a4141440 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a4143230 con 0x7fe9a409ac40 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a40b1a50 con 0x7fe9a41d74b0 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9aa97b640 10 monclient(hunting): authenticate will time out at 814.862517s 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a37fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe9a40b0fb0 0x7fe9a4139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55600/0 (socket says 172.21.15.117:55600) 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a37fe640 1 -- 172.21.15.117:0/853088781 learned_addr learned my addr 172.21.15.117:0/853088781 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a8f18640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe9a41d74b0 0x7fe9a41e25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40824/0 (socket says 172.21.15.117:40824) 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a3fff640 1 --1- 172.21.15.117:0/853088781 >> v1:172.21.15.173:6789/0 conn(0x7fe9a409ac40 0x7fe9a41d6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:41346/0 (socket says 172.21.15.117:41346) 2024-03-29T11:19:06.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9a40b0fb0 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9a41d74b0 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9a409ac40 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310837554 0 0) 0x7fe9a40b5160 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990004eb0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968274613 0 0) 0x7fe9a40b1a50 con 0x7fe9a41d74b0 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a40b5160 con 0x7fe9a41d74b0 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503467373 0 0) 0x7fe9a4143230 con 0x7fe9a409ac40 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.553+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a40b1a50 con 0x7fe9a409ac40 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4240603895 0 0) 0x7fe990004eb0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a4143230 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe994001960 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073651724 0 0) 0x7fe9a40b5160 con 0x7fe9a41d74b0 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe990004eb0 con 0x7fe9a41d74b0 2024-03-29T11:19:06.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe99c001ca0 con 0x7fe9a41d74b0 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.546 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3006230639 0 0) 0x7fe9a40b1a50 con 0x7fe9a409ac40 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a40b5160 con 0x7fe9a409ac40 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2219255569 0 0) 0x7fe9a4143230 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 >> v1:172.21.15.117:6790/0 conn(0x7fe9a41d74b0 legacy=0x7fe9a41e25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 >> v1:172.21.15.173:6789/0 conn(0x7fe9a409ac40 legacy=0x7fe9a41d6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 monclient: found mon.a 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a41461d0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:19:06.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 10 monclient: _renew_subs 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a413a430 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 10 monclient: _renew_subs 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a41428f0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9aa97b640 1 librados: init done 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe994004940 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe994004c50 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a27fc640 4 set_mon_vals no callback set 2024-03-29T11:19:06.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 monclient: dump: 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe994005140 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fe994005ac0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 10 client.4191.objecter ms_dispatch 0x7fe9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:06.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.554+0000 7fe9a0ff9640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.555+0000 7fe9aa97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.555+0000 7fe9aa97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.555+0000 7fe9aa97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.555+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe984005080 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.557+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe994005d50 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.557+0000 7fe9a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:06.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.557+0000 7fe9a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.616+0000 7fe9aa97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9aa97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9aa97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe984005330 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9a0ff9640 1 -- 172.21.15.117:0/853088781 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fe9940054a0 con 0x7fe9a40b0fb0 2024-03-29T11:19:06.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:06.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.617+0000 7fe9a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:06.595 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-29T11:19:06.595 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 10 librados: watch_flush enter 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 10 librados: watch_flush exit 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 10 monclient: shutdown 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:06.596 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 >> v1:172.21.15.117:6789/0 conn(0x7fe9a40b0fb0 legacy=0x7fe9a4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 shutdown_connections 2024-03-29T11:19:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 >> 172.21.15.117:0/853088781 conn(0x7fe9a4066b00 msgr2=0x7fe9a40b7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 shutdown_connections 2024-03-29T11:19:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 -- 172.21.15.117:0/853088781 wait complete. 2024-03-29T11:19:06.597 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-29T11:19:06.618+0000 7fe9aa97b640 1 librados: shutdown 2024-03-29T11:19:06.605 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:06.606 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:06.609 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.632+0000 7f7664248040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:06.617 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.640+0000 7f16bc410040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:06.704 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.727+0000 7f7664248040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:06.710 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.733+0000 7f16bc410040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:06.857 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.880+0000 7f7664248040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:06.867 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.890+0000 7f16bc410040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:06.899 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:06.922+0000 7f7664248040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:06.909 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:06.932+0000 7f16bc410040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:06.991 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-29T11:19:07.013+0000 7f4d5d613640 -1 Falling back to public interface 2024-03-29T11:19:06.994 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-29T11:19:07.017+0000 7fedd4393640 -1 Falling back to public interface 2024-03-29T11:19:07.006 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-03-29T11:19:07.029+0000 7fd336cb5640 -1 Falling back to public interface 2024-03-29T11:19:07.006 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-03-29T11:19:07.030+0000 7fae92393640 -1 Falling back to public interface 2024-03-29T11:19:07.011 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-03-29T11:19:07.034+0000 7f9b59268640 -1 Falling back to public interface 2024-03-29T11:19:07.014 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-29T11:19:07.037+0000 7fd020e0d640 -1 Falling back to public interface 2024-03-29T11:19:07.015 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-03-29T11:19:07.038+0000 7f381000c640 -1 Falling back to public interface 2024-03-29T11:19:07.018 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-29T11:19:07.041+0000 7f23b756d640 -1 Falling back to public interface 2024-03-29T11:19:07.057 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:07.080+0000 7f7664248040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:07.065 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:07.088+0000 7f16bc410040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:07.412 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.526+0000 7f0de3b15640 10 monclient: get_monmap_and_config 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.526+0000 7f0de3b15640 10 monclient: build_initial_monmap 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.526+0000 7f0de3b15640 10 monclient: monmap: 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:07.527319+0000 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:07.527319+0000 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:07.503 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.504 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:07.504 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:07.504 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:07.504 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.526+0000 7f0de3b15640 1 Processor -- start 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 1 -- start start 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient: init 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc0b0770 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0ad850 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc13c1b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:07.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0ac8e0 con 0x7f0ddc0ad850 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0ad330 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0acfe0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de3b15640 10 monclient(hunting): authenticate will time out at 815.836613s 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de10b0640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0ddc0ad850 0x7f0ddc0aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55608/0 (socket says 172.21.15.117:55608) 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de18b1640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0ddc0b0770 0x7f0ddc072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40830/0 (socket says 172.21.15.117:40830) 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de10b0640 1 -- 172.21.15.117:0/4147510040 learned_addr learned my addr 172.21.15.117:0/4147510040 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695752143 0 0) 0x7f0ddc0ac8e0 con 0x7f0ddc0ad850 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc8003520 con 0x7f0ddc0ad850 2024-03-29T11:19:07.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.527+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938174020 0 0) 0x7f0ddc0acfe0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc0ac8e0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764140125 0 0) 0x7f0ddc0ad330 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc0acfe0 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541824004 0 0) 0x7f0ddc0ac8e0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 >> v1:172.21.15.173:6789/0 conn(0x7f0ddc13c1b0 legacy=0x7f0ddc14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 >> v1:172.21.15.117:6789/0 conn(0x7f0ddc0ad850 legacy=0x7f0ddc0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: _finish_hunting 0 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 monclient: found mon.noname-b 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc0ab1f0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: _finish_auth 0 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: _check_auth_tickets 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dd8001670 con 0x7f0ddc0b0770 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: dump: 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.515 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dd8001850 con 0x7f0ddc0b0770 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 1 -- 172.21.15.117:0/4147510040 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dd8001ca0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 4 set_mon_vals no callback set 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de08af640 10 monclient: dump: 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 10 monclient: shutdown 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 1 -- 172.21.15.117:0/4147510040 >> v1:172.21.15.117:6790/0 conn(0x7f0ddc0b0770 legacy=0x7f0ddc072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 1 -- 172.21.15.117:0/4147510040 shutdown_connections 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.528+0000 7f0de3b15640 1 -- 172.21.15.117:0/4147510040 >> 172.21.15.117:0/4147510040 conn(0x7f0ddc06b660 msgr2=0x7f0ddc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 1 -- 172.21.15.117:0/4147510040 shutdown_connections 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 1 -- 172.21.15.117:0/4147510040 wait complete. 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 10 monclient: build_initial_monmap 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 10 monclient: monmap: 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:07.518 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 1 librados: starting msgr at 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.529+0000 7f0de3b15640 1 librados: starting objecter 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 Processor -- start 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 -- start start 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 librados: setting wanted keys 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 librados: calling monclient init 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient: init 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:07.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc0ad850 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0b0770 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc13c1b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc137970 con 0x7f0ddc0b0770 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0b30e0 con 0x7f0ddc0ad850 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc137690 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de3b15640 10 monclient(hunting): authenticate will time out at 815.839612s 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de10b0640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0ddc0b0770 0x7f0ddc137d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55614/0 (socket says 172.21.15.117:55614) 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de20b2640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0ddc13c1b0 0x7f0ddc1384a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40838/0 (socket says 172.21.15.117:40838) 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0de10b0640 1 -- 172.21.15.117:0/78367208 learned_addr learned my addr 172.21.15.117:0/78367208 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc13c1b0 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc0b0770 2024-03-29T11:19:07.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc0ad850 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020350637 0 0) 0x7f0ddc137970 con 0x7f0ddc0b0770 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc0004eb0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804274942 0 0) 0x7f0ddc137690 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.530+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc137970 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902610674 0 0) 0x7f0ddc0b30e0 con 0x7f0ddc0ad850 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): my global_id is 4198 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc137690 con 0x7f0ddc0ad850 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314556786 0 0) 0x7f0dc0004eb0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ddc0b30e0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dcc0013b0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): dump: 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.522 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2988105731 0 0) 0x7f0ddc137970 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dc0004eb0 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4016660062 0 0) 0x7f0ddc137690 con 0x7f0ddc0ad850 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ddc137970 con 0x7f0ddc0ad850 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dd4001b50 con 0x7f0ddc13c1b0 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): dump: 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dd8001b60 con 0x7f0ddc0ad850 2024-03-29T11:19:07.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient(hunting): dump: 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 776047078 0 0) 0x7f0ddc0b30e0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 >> v1:172.21.15.117:6790/0 conn(0x7f0ddc13c1b0 legacy=0x7f0ddc1384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 >> v1:172.21.15.173:6789/0 conn(0x7f0ddc0ad850 legacy=0x7f0ddc0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 monclient: found mon.a 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc1351b0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 5 monclient: authenticate success, global_id 4197 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 10 monclient: _renew_subs 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc1367c0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:07.526 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 10 monclient: _renew_subs 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc138d90 con 0x7f0ddc0b0770 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0de3b15640 1 librados: init done 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dcc001590 con 0x7f0ddc0b0770 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dcc004c30 con 0x7f0ddc0b0770 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd3fff640 4 set_mon_vals no callback set 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 monclient: dump: 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.527 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0dcc005120 con 0x7f0ddc0b0770 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f0dcc006a30 con 0x7f0ddc0b0770 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 10 client.4197.objecter ms_dispatch 0x7f0ddc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.531+0000 7f0dd27fc640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.532+0000 7f0de3b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:07.528 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.532+0000 7f0de3b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:07.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.532+0000 7f0de3b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:07.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.532+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc4005080 con 0x7f0ddc0b0770 2024-03-29T11:19:07.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.534+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0dcc006470 con 0x7f0ddc0b0770 2024-03-29T11:19:07.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.534+0000 7f0dd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:07.529 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.534+0000 7f0dd27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:07.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.594+0000 7f0de3b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0de3b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0de3b15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:07.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dc4005330 con 0x7f0ddc0b0770 2024-03-29T11:19:07.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0dd27fc640 1 -- 172.21.15.117:0/78367208 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f0dcc0066c0 con 0x7f0ddc0b0770 2024-03-29T11:19:07.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0dd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:07.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.595+0000 7f0dd27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:07.573 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:07.573 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 10 librados: watch_flush enter 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 10 librados: watch_flush exit 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 10 client.4197.objecter successfully canceled tick 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 10 monclient: shutdown 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 >> v1:172.21.15.117:6789/0 conn(0x7f0ddc0b0770 legacy=0x7f0ddc137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 shutdown_connections 2024-03-29T11:19:07.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 >> 172.21.15.117:0/78367208 conn(0x7f0ddc06b660 msgr2=0x7f0ddc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:07.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 shutdown_connections 2024-03-29T11:19:07.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 -- 172.21.15.117:0/78367208 wait complete. 2024-03-29T11:19:07.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:07.596+0000 7f0de3b15640 1 librados: shutdown 2024-03-29T11:19:07.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:07.869 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:07.892+0000 7f7664248040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:07.882 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:07.905+0000 7f16bc410040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:08.145 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:08.168+0000 7f7664248040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:08.158 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:08.181+0000 7f16bc410040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:08.196 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:08.219+0000 7f7664248040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:08.208 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:08.231+0000 7f16bc410040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:08.310 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:08.333+0000 7f7664248040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:08.325 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T11:19:08.348+0000 7f16bc410040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:09.324 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:09.347+0000 7f7604f11640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:09.324 INFO:tasks.ceph.mgr.y.smithi117.stderr:2024-03-29T11:19:09.347+0000 7f7604f11640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:10.462 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-29T11:19:10.485+0000 7fedd4393640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:10.492 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-29T11:19:10.515+0000 7f4d5d613640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:10.493 INFO:tasks.ceph.osd.3.smithi117.stderr:2024-03-29T11:19:10.516+0000 7fae92393640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:10.521 INFO:tasks.ceph.osd.0.smithi117.stderr:2024-03-29T11:19:10.544+0000 7fd336cb5640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:10.527 INFO:tasks.ceph.osd.2.smithi117.stderr:2024-03-29T11:19:10.550+0000 7f9b59268640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:10.531 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-29T11:19:10.554+0000 7fd020e0d640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:10.533 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-29T11:19:10.556+0000 7f23b756d640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:10.535 INFO:tasks.ceph.osd.1.smithi117.stderr:2024-03-29T11:19:10.558+0000 7f381000c640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:14.388 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.560+0000 7f0db8a35640 10 monclient: get_monmap_and_config 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.560+0000 7f0db8a35640 10 monclient: build_initial_monmap 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.560+0000 7f0db8a35640 10 monclient: monmap: 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:14.537 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:14.561537+0000 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:14.561537+0000 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.561+0000 7f0db8a35640 1 Processor -- start 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.561+0000 7f0db8a35640 1 -- start start 2024-03-29T11:19:14.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.561+0000 7f0db8a35640 10 monclient: init 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-c con 0x7f0db41457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-a con 0x7f0db413f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-b con 0x7f0db40a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db40aaef0 con 0x7f0db413f650 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4145bd0 con 0x7f0db41457f0 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4145eb0 con 0x7f0db40a9d70 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db8a35640 10 monclient(hunting): authenticate will time out at 822.871812s 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db1d9b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0db413f650 0x7f0db40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:38000/0 (socket says 172.21.15.117:38000) 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.562+0000 7f0db1d9b640 1 -- 172.21.15.117:0/880709024 learned_addr learned my addr 172.21.15.117:0/880709024 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841386892 0 0) 0x7f0db4145bd0 con 0x7f0db41457f0 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c003520 con 0x7f0db41457f0 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846146492 0 0) 0x7f0db40aaef0 con 0x7f0db413f650 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4145bd0 con 0x7f0db413f650 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766827132 0 0) 0x7f0db4145eb0 con 0x7f0db40a9d70 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.563+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c0039d0 con 0x7f0db40a9d70 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947627876 0 0) 0x7f0d9c003520 con 0x7f0db41457f0 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 >> v1:172.21.15.117:6790/0 conn(0x7f0db40a9d70 legacy=0x7f0db414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 >> v1:172.21.15.117:6789/0 conn(0x7f0db413f650 legacy=0x7f0db40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 monclient: found mon.noname-c 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4146340 con 0x7f0db41457f0 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: _finish_auth 0 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:14.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0da4001690 con 0x7f0db41457f0 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: dump: 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db8a35640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:19:14.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db8a35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0da40017f0 con 0x7f0db41457f0 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 1 -- 172.21.15.117:0/880709024 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0da4001c00 con 0x7f0db41457f0 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db8a35640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db8a35640 4 set_mon_vals no callback set 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.564+0000 7f0db159a640 10 monclient: dump: 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.547 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 10 monclient: shutdown 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 1 -- 172.21.15.117:0/880709024 >> v1:172.21.15.173:6789/0 conn(0x7f0db41457f0 legacy=0x7f0db413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 1 -- 172.21.15.117:0/880709024 shutdown_connections 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 1 -- 172.21.15.117:0/880709024 >> 172.21.15.117:0/880709024 conn(0x7f0db4066a00 msgr2=0x7f0db4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 1 -- 172.21.15.117:0/880709024 shutdown_connections 2024-03-29T11:19:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.565+0000 7f0db8a35640 1 -- 172.21.15.117:0/880709024 wait complete. 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.566+0000 7f0db8a35640 10 monclient: build_initial_monmap 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.566+0000 7f0db8a35640 10 monclient: monmap: 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 librados: starting msgr at 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 librados: starting objecter 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 Processor -- start 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 -- start start 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 librados: setting wanted keys 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 1 librados: calling monclient init 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.567+0000 7f0db8a35640 10 monclient: init 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-a con 0x7f0db40a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-c con 0x7f0db413f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient(hunting): picked mon.noname-b con 0x7f0db41457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4130760 con 0x7f0db40a9d70 2024-03-29T11:19:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4070000 con 0x7f0db41457f0 2024-03-29T11:19:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db4070270 con 0x7f0db413f650 2024-03-29T11:19:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db8a35640 10 monclient(hunting): authenticate will time out at 822.877532s 2024-03-29T11:19:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db259c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0db40a9d70 0x7f0db4140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:38136/0 (socket says 172.21.15.117:38136) 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0db259c640 1 -- 172.21.15.117:0/2804530589 learned_addr learned my addr 172.21.15.117:0/2804530589 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0da2ffd640 10 client.?.objecter ms_handle_connect 0x7f0db41457f0 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0da2ffd640 10 client.?.objecter ms_handle_connect 0x7f0db40a9d70 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.568+0000 7f0da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 client.?.objecter ms_handle_connect 0x7f0db413f650 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587842278 0 0) 0x7f0db4070000 con 0x7f0db41457f0 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d94004eb0 con 0x7f0db41457f0 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373797193 0 0) 0x7f0db4130760 con 0x7f0db40a9d70 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:14.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4070000 con 0x7f0db40a9d70 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179963761 0 0) 0x7f0db4070270 con 0x7f0db413f650 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4130760 con 0x7f0db413f650 2024-03-29T11:19:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3803162083 0 0) 0x7f0d94004eb0 con 0x7f0db41457f0 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0db4070270 con 0x7f0db41457f0 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dac001bc0 con 0x7f0db41457f0 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:14.562 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.563 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1063778968 0 0) 0x7f0db4070270 con 0x7f0db41457f0 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.569+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 >> v1:172.21.15.117:6790/0 conn(0x7f0db413f650 legacy=0x7f0db4070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 >> v1:172.21.15.117:6789/0 conn(0x7f0db40a9d70 legacy=0x7f0db4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 1 monclient: found mon.b 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db413c6d0 con 0x7f0db41457f0 2024-03-29T11:19:14.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 10 monclient: _renew_subs 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db4142d60 con 0x7f0db41457f0 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 10 monclient: _renew_subs 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db406f230 con 0x7f0db41457f0 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db8a35640 1 librados: init done 2024-03-29T11:19:14.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dac0020d0 con 0x7f0db41457f0 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0dac003b30 con 0x7f0db41457f0 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0da2ffd640 10 monclient: dump: 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:14.568 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:14.569 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.570+0000 7f0db0d99640 4 set_mon_vals no callback set 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.571+0000 7f0db8a35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.571+0000 7f0db8a35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.571+0000 7f0db8a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.572+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db413c910 con 0x7f0db41457f0 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.572+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f0dac019b00 con 0x7f0db41457f0 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f0dac083bb0 con 0x7f0db41457f0 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 10 client.4243.objecter ms_dispatch 0x7f0db4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0db259c640 10 monclient: get_auth_request con 0x7f0d9406de50 auth_method 0 2024-03-29T11:19:14.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:19:14.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.573+0000 7f0da2ffd640 10 client.4243.objecter ms_handle_connect 0x7f0d9406de50 2024-03-29T11:19:14.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.579+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0dac04f2c0 con 0x7f0db41457f0 2024-03-29T11:19:14.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.579+0000 7f0da2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:14.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.579+0000 7f0da2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:14.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.975+0000 7f0db8a35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.975+0000 7f0db8a35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.975+0000 7f0db8a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:19:14.952 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.975+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0db4075f10 con 0x7f0db41457f0 2024-03-29T11:19:14.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.976+0000 7f0da2ffd640 1 -- 172.21.15.117:0/2804530589 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f0dac0513c0 con 0x7f0db41457f0 2024-03-29T11:19:14.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.976+0000 7f0da2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:14.953 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.976+0000 7f0da2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:14.954 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:14.954 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.979+0000 7f0db8a35640 10 librados: watch_flush enter 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 10 librados: watch_flush exit 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 >> v1:172.21.15.117:6816/4276038175 conn(0x7f0d9406de50 legacy=0x7f0d94070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 10 monclient: shutdown 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 >> v1:172.21.15.173:6789/0 conn(0x7f0db41457f0 legacy=0x7f0db41e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 shutdown_connections 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 >> 172.21.15.117:0/2804530589 conn(0x7f0db4066a00 msgr2=0x7f0db40b6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:14.957 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 shutdown_connections 2024-03-29T11:19:14.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 -- 172.21.15.117:0/2804530589 wait complete. 2024-03-29T11:19:14.958 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:14.980+0000 7f0db8a35640 1 librados: shutdown 2024-03-29T11:19:14.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:21.778 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:21.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.948+0000 7f36c110e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:21.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.948+0000 7f36c110e640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.948+0000 7f36c110e640 10 monclient: monmap: 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:21.949854+0000 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:21.949854+0000 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:21.926 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.950+0000 7f36c110e640 1 Processor -- start 2024-03-29T11:19:21.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.950+0000 7f36c110e640 1 -- start start 2024-03-29T11:19:21.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.950+0000 7f36c110e640 10 monclient: init 2024-03-29T11:19:21.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36c110e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.927 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36c110e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-b con 0x7f36bc1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-a con 0x7f36bc13f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-c con 0x7f36bc0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36bad9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f36bc1457f0 0x7f36bc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51360/0 (socket says 172.21.15.117:51360) 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.951+0000 7f36bad9d640 1 -- 172.21.15.117:0/669449873 learned_addr learned my addr 172.21.15.117:0/669449873 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0aaef0 con 0x7f36bc13f650 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc145bd0 con 0x7f36bc0a9d70 2024-03-29T11:19:21.929 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc145eb0 con 0x7f36bc1457f0 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36c110e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36c110e640 10 monclient(hunting): authenticate will time out at 830.261399s 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164311360 0 0) 0x7f36bc0aaef0 con 0x7f36bc13f650 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:21.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.952+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a8003520 con 0x7f36bc13f650 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246908321 0 0) 0x7f36bc145eb0 con 0x7f36bc1457f0 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:19:21.932 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc0aaef0 con 0x7f36bc1457f0 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118688776 0 0) 0x7f36a8003520 con 0x7f36bc13f650 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 >> v1:172.21.15.117:6790/0 conn(0x7f36bc1457f0 legacy=0x7f36bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 >> v1:172.21.15.173:6789/0 conn(0x7f36bc0a9d70 legacy=0x7f36bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 monclient: found mon.noname-a 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc146340 con 0x7f36bc13f650 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36a4001690 con 0x7f36bc13f650 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36c110e640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:19:21.933 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36c110e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: dump: 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.934 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a40018b0 con 0x7f36bc13f650 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36c110e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36ba59c640 1 -- 172.21.15.117:0/669449873 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36a4001c40 con 0x7f36bc13f650 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36c110e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.953+0000 7f36c110e640 4 set_mon_vals no callback set 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36c110e640 10 monclient: shutdown 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36ba59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36ba59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36ba59c640 10 monclient: dump: 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:21.935 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36c110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 >> v1:172.21.15.117:6789/0 conn(0x7f36bc13f650 legacy=0x7f36bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 shutdown_connections 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.954+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 >> 172.21.15.117:0/669449873 conn(0x7f36bc066a00 msgr2=0x7f36bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.955+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 shutdown_connections 2024-03-29T11:19:21.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.955+0000 7f36c110e640 1 -- 172.21.15.117:0/669449873 wait complete. 2024-03-29T11:19:21.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.956+0000 7f36c110e640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.956+0000 7f36c110e640 10 monclient: monmap: 2024-03-29T11:19:21.939 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.956+0000 7f36c110e640 1 librados: starting msgr at 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 1 librados: starting objecter 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 1 Processor -- start 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 1 -- start start 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 1 librados: setting wanted keys 2024-03-29T11:19:21.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 1 librados: calling monclient init 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.957+0000 7f36c110e640 10 monclient: init 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-b con 0x7f36bc0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-a con 0x7f36bc13f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36bad9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f36bc0a9d70 0x7f36bc0ab870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:42384/0 (socket says 172.21.15.117:42384) 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36bad9d640 1 -- 172.21.15.117:0/3109464296 learned_addr learned my addr 172.21.15.117:0/3109464296 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 10 monclient(hunting): picked mon.noname-c con 0x7f36bc1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0ad190 con 0x7f36bc13f650 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0ac690 con 0x7f36bc0a9d70 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36b37fe640 10 client.?.objecter ms_handle_connect 0x7f36bc13f650 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36b37fe640 10 client.?.objecter ms_handle_connect 0x7f36bc0a9d70 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.958+0000 7f36b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 client.?.objecter ms_handle_connect 0x7f36bc1457f0 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36bc0ad3e0 con 0x7f36bc1457f0 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36c110e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36c110e640 10 monclient(hunting): authenticate will time out at 830.268198s 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982993926 0 0) 0x7f36bc0ad190 con 0x7f36bc13f650 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f369c004eb0 con 0x7f36bc13f650 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073210924 0 0) 0x7f36bc0ad3e0 con 0x7f36bc1457f0 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc0ad190 con 0x7f36bc1457f0 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2782213090 0 0) 0x7f369c004eb0 con 0x7f36bc13f650 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36bc0ad3e0 con 0x7f36bc13f650 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36a4001610 con 0x7f36bc13f650 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.943 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201208220 0 0) 0x7f36bc0ad190 con 0x7f36bc1457f0 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f369c004eb0 con 0x7f36bc1457f0 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.959+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36b4001b10 con 0x7f36bc1457f0 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient(hunting): dump: 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.944 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1580704245 0 0) 0x7f369c004eb0 con 0x7f36bc1457f0 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 >> v1:172.21.15.173:6789/0 conn(0x7f36bc0a9d70 legacy=0x7f36bc0ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 >> v1:172.21.15.117:6789/0 conn(0x7f36bc13f650 legacy=0x7f36bc0abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 monclient: found mon.c 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:21.945 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc1411f0 con 0x7f36bc1457f0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 10 monclient: _renew_subs 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36bc137c30 con 0x7f36bc1457f0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 10 monclient: _renew_subs 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc142a70 con 0x7f36bc1457f0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36c110e640 1 librados: init done 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b4002080 con 0x7f36bc1457f0 2024-03-29T11:19:21.946 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.960+0000 7f36b9d9b640 4 set_mon_vals no callback set 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.961+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36b4003b10 con 0x7f36bc1457f0 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.961+0000 7f36b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.961+0000 7f36b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.961+0000 7f36b37fe640 10 monclient: dump: 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:21.947 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:21.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.962+0000 7f36c110e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.962+0000 7f36c110e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.962+0000 7f36c110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:21.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.962+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a0005080 con 0x7f36bc1457f0 2024-03-29T11:19:21.948 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.963+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f36b4019b20 con 0x7f36bc1457f0 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f36b4083ad0 con 0x7f36bc1457f0 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 10 client.4241.objecter ms_dispatch 0x7f36bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36bad9d640 10 monclient: get_auth_request con 0x7f369c06dee0 auth_method 0 2024-03-29T11:19:21.949 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.964+0000 7f36b37fe640 10 client.4241.objecter ms_handle_connect 0x7f369c06dee0 2024-03-29T11:19:21.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.974+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36b404f410 con 0x7f36bc1457f0 2024-03-29T11:19:21.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.974+0000 7f36b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.951 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:21.974+0000 7f36b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36c110e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36c110e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36c110e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36a0005600 con 0x7f36bc1457f0 2024-03-29T11:19:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36b37fe640 1 -- 172.21.15.117:0/3109464296 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f36b4083180 con 0x7f36bc1457f0 2024-03-29T11:19:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.337+0000 7f36b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:22.315 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:22.315 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:22.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.340+0000 7f36c110e640 10 librados: watch_flush enter 2024-03-29T11:19:22.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 10 librados: watch_flush exit 2024-03-29T11:19:22.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T11:19:22.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 10 client.4241.objecter successfully canceled tick 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 >> v1:172.21.15.117:6816/4276038175 conn(0x7f369c06dee0 legacy=0x7f369c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 10 monclient: shutdown 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 >> v1:172.21.15.117:6790/0 conn(0x7f36bc1457f0 legacy=0x7f36bc1e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 shutdown_connections 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 >> 172.21.15.117:0/3109464296 conn(0x7f36bc066a00 msgr2=0x7f36bc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 shutdown_connections 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 -- 172.21.15.117:0/3109464296 wait complete. 2024-03-29T11:19:22.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:22.341+0000 7f36c110e640 1 librados: shutdown 2024-03-29T11:19:22.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:29.139 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:29.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.307+0000 7fd9d0874640 10 monclient: get_monmap_and_config 2024-03-29T11:19:29.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.307+0000 7fd9d0874640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.307+0000 7fd9d0874640 10 monclient: monmap: 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:29.308842+0000 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:29.308842+0000 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:29.285 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.309+0000 7fd9d0874640 1 Processor -- start 2024-03-29T11:19:29.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.309+0000 7fd9d0874640 1 -- start start 2024-03-29T11:19:29.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.309+0000 7fd9d0874640 10 monclient: init 2024-03-29T11:19:29.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:29.288 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-a con 0x7fd9cc1457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-c con 0x7fd9cc13f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-b con 0x7fd9cc0a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc0aaef0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc145bd0 con 0x7fd9cc13f650 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc145eb0 con 0x7fd9cc0a9d70 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.289 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.310+0000 7fd9d0874640 10 monclient(hunting): authenticate will time out at 837.619898s 2024-03-29T11:19:29.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.313+0000 7fd9cad9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd9cc0a9d70 0x7fd9cc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:54759/0 (socket says 172.21.15.117:54759) 2024-03-29T11:19:29.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.313+0000 7fd9cad9d640 1 -- 172.21.15.117:0/1030296200 learned_addr learned my addr 172.21.15.117:0/1030296200 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:29.290 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826713481 0 0) 0x7fd9cc145eb0 con 0x7fd9cc0a9d70 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b4003520 con 0x7fd9cc0a9d70 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482373046 0 0) 0x7fd9cc145bd0 con 0x7fd9cc13f650 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:19:29.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9cc145eb0 con 0x7fd9cc13f650 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1647259158 0 0) 0x7fd9cc145eb0 con 0x7fd9cc13f650 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 >> v1:172.21.15.117:6790/0 conn(0x7fd9cc0a9d70 legacy=0x7fd9cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 >> v1:172.21.15.117:6789/0 conn(0x7fd9cc1457f0 legacy=0x7fd9cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 monclient: found mon.noname-c 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc146340 con 0x7fd9cc13f650 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9bc001670 con 0x7fd9cc13f650 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9c959a640 10 monclient: dump: 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9d0874640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.314+0000 7fd9d0874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9bc001850 con 0x7fd9cc13f650 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 1 -- 172.21.15.117:0/1030296200 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9bc001ca0 con 0x7fd9cc13f650 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9c959a640 10 monclient: dump: 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9d0874640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.316+0000 7fd9d0874640 4 set_mon_vals no callback set 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 10 monclient: shutdown 2024-03-29T11:19:29.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 1 -- 172.21.15.117:0/1030296200 >> v1:172.21.15.173:6789/0 conn(0x7fd9cc13f650 legacy=0x7fd9cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 1 -- 172.21.15.117:0/1030296200 shutdown_connections 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 1 -- 172.21.15.117:0/1030296200 >> 172.21.15.117:0/1030296200 conn(0x7fd9cc066a00 msgr2=0x7fd9cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 1 -- 172.21.15.117:0/1030296200 shutdown_connections 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.317+0000 7fd9d0874640 1 -- 172.21.15.117:0/1030296200 wait complete. 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.318+0000 7fd9d0874640 10 monclient: build_initial_monmap 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.318+0000 7fd9d0874640 10 monclient: monmap: 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:29.296 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 librados: starting msgr at 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 librados: starting objecter 2024-03-29T11:19:29.297 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 Processor -- start 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 -- start start 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 librados: setting wanted keys 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 1 librados: calling monclient init 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.319+0000 7fd9d0874640 10 monclient: init 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-a con 0x7fd9cc0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-b con 0x7fd9cc13f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient(hunting): picked mon.noname-c con 0x7fd9cc1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc142950 con 0x7fd9cc0a9d70 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc141e50 con 0x7fd9cc13f650 2024-03-29T11:19:29.298 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc141240 con 0x7fd9cc1457f0 2024-03-29T11:19:29.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:29.299 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.320+0000 7fd9d0874640 10 monclient(hunting): authenticate will time out at 837.629346s 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.323+0000 7fd9c9d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd9cc13f650 0x7fd9cc13a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:52358/0 (socket says 172.21.15.117:52358) 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.323+0000 7fd9c9d9b640 1 -- 172.21.15.117:0/1291207128 learned_addr learned my addr 172.21.15.117:0/1291207128 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9cc1457f0 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199812908 0 0) 0x7fd9cc141240 con 0x7fd9cc1457f0 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:29.302 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:29.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:19:29.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.324+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ac004eb0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.325+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9cc13f650 2024-03-29T11:19:29.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.325+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138176781 0 0) 0x7fd9ac004eb0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9cc141240 con 0x7fd9cc1457f0 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9c0001950 con 0x7fd9cc1457f0 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient(hunting): dump: 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.304 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9cc0a9d70 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2828142479 0 0) 0x7fd9cc141240 con 0x7fd9cc1457f0 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 >> v1:172.21.15.173:6789/0 conn(0x7fd9cc13f650 legacy=0x7fd9cc13a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 >> v1:172.21.15.117:6789/0 conn(0x7fd9cc0a9d70 legacy=0x7fd9cc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 monclient: found mon.c 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:29.305 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc142de0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 10 monclient: _renew_subs 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc1420c0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 10 monclient: _renew_subs 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc135a40 con 0x7fd9cc1457f0 2024-03-29T11:19:29.306 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.326+0000 7fd9d0874640 1 librados: init done 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.327+0000 7fd9d0874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.328+0000 7fd9d0874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.328+0000 7fd9d0874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.328+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc135c90 con 0x7fd9cc1457f0 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c0001d60 con 0x7fd9cc1457f0 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:29.307 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9c0004980 con 0x7fd9cc1457f0 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9baffd640 10 monclient: dump: 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.329+0000 7fd9bbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:29.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.331+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd9c0004ba0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fd9c0084a10 con 0x7fd9cc1457f0 2024-03-29T11:19:29.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 10 client.4247.objecter ms_dispatch 0x7fd9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:29.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:29.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:29.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:19:29.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9ca59c640 10 monclient: get_auth_request con 0x7fd9ac06de10 auth_method 0 2024-03-29T11:19:29.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.332+0000 7fd9baffd640 10 client.4247.objecter ms_handle_connect 0x7fd9ac06de10 2024-03-29T11:19:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.339+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9c0053eb0 con 0x7fd9cc1457f0 2024-03-29T11:19:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.339+0000 7fd9baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.339+0000 7fd9baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:29.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.655+0000 7fd9d0874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.655+0000 7fd9d0874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.655+0000 7fd9d0874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:29.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.655+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9cc075f10 con 0x7fd9cc1457f0 2024-03-29T11:19:29.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.656+0000 7fd9baffd640 1 -- 172.21.15.117:0/1291207128 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fd9c0052220 con 0x7fd9cc1457f0 2024-03-29T11:19:29.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.656+0000 7fd9baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:29.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.656+0000 7fd9baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:29.633 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:29.633 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:29.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.661+0000 7fd9d0874640 10 librados: watch_flush enter 2024-03-29T11:19:29.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.661+0000 7fd9d0874640 10 librados: watch_flush exit 2024-03-29T11:19:29.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.661+0000 7fd9d0874640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:19:29.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.661+0000 7fd9d0874640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd9ac06de10 legacy=0x7fd9ac0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 10 monclient: shutdown 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 >> v1:172.21.15.117:6790/0 conn(0x7fd9cc1457f0 legacy=0x7fd9cc1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 shutdown_connections 2024-03-29T11:19:29.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 >> 172.21.15.117:0/1291207128 conn(0x7fd9cc066a00 msgr2=0x7fd9cc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:29.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 shutdown_connections 2024-03-29T11:19:29.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 -- 172.21.15.117:0/1291207128 wait complete. 2024-03-29T11:19:29.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:29.662+0000 7fd9d0874640 1 librados: shutdown 2024-03-29T11:19:29.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:36.463 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:36.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.636+0000 7f7fb2f0e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:36.614 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.636+0000 7f7fb2f0e640 10 monclient: build_initial_monmap 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.636+0000 7f7fb2f0e640 10 monclient: monmap: 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:36.637269+0000 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:36.637269+0000 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.637+0000 7f7fb2f0e640 1 Processor -- start 2024-03-29T11:19:36.615 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 1 -- start start 2024-03-29T11:19:36.616 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 10 monclient: init 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fac0a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.638+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fac13ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fac0af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:36.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0afd10 con 0x7f7fac13ee90 2024-03-29T11:19:36.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0af6f0 con 0x7f7fac0af900 2024-03-29T11:19:36.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0aac80 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:36.621 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.639+0000 7f7fb2f0e640 10 monclient(hunting): authenticate will time out at 844.948044s 2024-03-29T11:19:36.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.645+0000 7f7fabfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7fac13ee90 0x7f7fac13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59331/0 (socket says 172.21.15.117:59331) 2024-03-29T11:19:36.622 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.645+0000 7f7fabfff640 1 -- 172.21.15.117:0/3842825104 learned_addr learned my addr 172.21.15.117:0/3842825104 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:36.623 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335433575 0 0) 0x7f7fac0afd10 con 0x7f7fac13ee90 2024-03-29T11:19:36.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:19:36.624 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.646+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f94003520 con 0x7f7fac13ee90 2024-03-29T11:19:36.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781652851 0 0) 0x7f7f94003520 con 0x7f7fac13ee90 2024-03-29T11:19:36.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 >> v1:172.21.15.117:6790/0 conn(0x7f7fac0a9ac0 legacy=0x7f7fac0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 >> v1:172.21.15.173:6789/0 conn(0x7f7fac0af900 legacy=0x7f7fac149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0aaf60 con 0x7f7fac13ee90 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7fa40019f0 con 0x7f7fac13ee90 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.649+0000 7f7fab7fe640 10 monclient: dump: 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fb2f0e640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:19:36.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fb2f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:36.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa4001bc0 con 0x7f7fac13ee90 2024-03-29T11:19:36.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:36.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fb2f0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.650+0000 7f7fb2f0e640 4 set_mon_vals no callback set 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fab7fe640 1 -- 172.21.15.117:0/3842825104 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7fa4001fd0 con 0x7f7fac13ee90 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fab7fe640 10 monclient: dump: 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fb2f0e640 10 monclient: shutdown 2024-03-29T11:19:36.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fb2f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:36.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.651+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/3842825104 >> v1:172.21.15.117:6789/0 conn(0x7f7fac13ee90 legacy=0x7f7fac13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.652+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/3842825104 shutdown_connections 2024-03-29T11:19:36.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.652+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/3842825104 >> 172.21.15.117:0/3842825104 conn(0x7f7fac0b35e0 msgr2=0x7f7fac0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.653+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/3842825104 shutdown_connections 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.653+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/3842825104 wait complete. 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 10 monclient: build_initial_monmap 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 10 monclient: monmap: 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.633 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 librados: starting msgr at 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 librados: starting objecter 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 Processor -- start 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 -- start start 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 librados: setting wanted keys 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 1 librados: calling monclient init 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.654+0000 7f7fb2f0e640 10 monclient: init 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fac0a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:36.634 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fac0af900 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fac13ee90 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac140d60 con 0x7f7fac0af900 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac13de20 con 0x7f7fac13ee90 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac141fa0 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:36.635 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.655+0000 7f7fb2f0e640 10 monclient(hunting): authenticate will time out at 844.964882s 2024-03-29T11:19:36.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fabfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7fac0af900 0x7f7fac13c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59431/0 (socket says 172.21.15.117:59431) 2024-03-29T11:19:36.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fabfff640 1 -- 172.21.15.117:0/1770945547 learned_addr learned my addr 172.21.15.117:0/1770945547 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fb14ab640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7fac13ee90 0x7f7fac200cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:33557/0 (socket says 172.21.15.117:33557) 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fa97fa640 10 client.?.objecter ms_handle_connect 0x7f7fac0af900 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fa97fa640 10 client.?.objecter ms_handle_connect 0x7f7fac13ee90 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.661+0000 7f7fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645756659 0 0) 0x7f7fac140d60 con 0x7f7fac0af900 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98004eb0 con 0x7f7fac0af900 2024-03-29T11:19:36.641 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462871178 0 0) 0x7f7fac13de20 con 0x7f7fac13ee90 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98006230 con 0x7f7fac13ee90 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 client.?.objecter ms_handle_connect 0x7f7fac0a9ac0 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.662+0000 7f7fa97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848192065 0 0) 0x7f7fac141fa0 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:36.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f98006570 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1621736341 0 0) 0x7f7f98004eb0 con 0x7f7fac0af900 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fac141fa0 con 0x7f7fac0af900 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7fa4001fd0 con 0x7f7fac0af900 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.643 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957141398 0 0) 0x7f7f98006570 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f98006eb0 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7f9c0018b0 con 0x7f7fac0a9ac0 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.663+0000 7f7fa97fa640 10 monclient(hunting): dump: 2024-03-29T11:19:36.644 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:36.645 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776652757 0 0) 0x7f7fac141fa0 con 0x7f7fac0af900 2024-03-29T11:19:36.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 >> v1:172.21.15.117:6790/0 conn(0x7f7fac0a9ac0 legacy=0x7f7fac13d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 >> v1:172.21.15.173:6789/0 conn(0x7f7fac13ee90 legacy=0x7f7fac200cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 1 monclient: found mon.a 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac142280 con 0x7f7fac0af900 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fa97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 10 monclient: _renew_subs 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac142560 con 0x7f7fac0af900 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:36.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 10 monclient: _renew_subs 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac13cf40 con 0x7f7fac0af900 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.664+0000 7f7fb2f0e640 1 librados: init done 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fb2f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fb2f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fb2f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac13cf40 con 0x7f7fac0af900 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa4004e10 con 0x7f7fac0af900 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7fa97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.666+0000 7f7faaffd640 4 set_mon_vals no callback set 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.668+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7fa4001fd0 con 0x7f7fac0af900 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.668+0000 7f7fa97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.668+0000 7f7fa97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.668+0000 7f7fa97fa640 10 monclient: dump: 2024-03-29T11:19:36.648 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:36.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.669+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7fa401adf0 con 0x7f7fac0af900 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.670+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f7fa4054650 con 0x7f7fac0af900 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.670+0000 7f7fa97fa640 10 client.4260.objecter ms_dispatch 0x7f7fac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.670+0000 7f7fa97fa640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.670+0000 7f7fa97fa640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.670+0000 7f7fa97fa640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:19:36.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.673+0000 7f7fb0caa640 10 monclient: get_auth_request con 0x7f7f9806ea60 auth_method 0 2024-03-29T11:19:36.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.674+0000 7f7fa97fa640 10 client.4260.objecter ms_handle_connect 0x7f7f9806ea60 2024-03-29T11:19:36.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.677+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fa40508f0 con 0x7f7fac0af900 2024-03-29T11:19:36.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.677+0000 7f7fa97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:36.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:36.677+0000 7f7fa97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:36.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.003+0000 7f7fb2f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.003+0000 7f7fb2f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.003+0000 7f7fb2f0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:19:36.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.003+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fac13cf40 con 0x7f7fac0af900 2024-03-29T11:19:36.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.007+0000 7f7fa97fa640 1 -- 172.21.15.117:0/1770945547 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f7fa40508f0 con 0x7f7fac0af900 2024-03-29T11:19:36.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.007+0000 7f7fa97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:36.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.007+0000 7f7fa97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:36.984 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:36.984 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:36.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.011+0000 7f7fb2f0e640 10 librados: watch_flush enter 2024-03-29T11:19:36.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.011+0000 7f7fb2f0e640 10 librados: watch_flush exit 2024-03-29T11:19:36.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.011+0000 7f7fb2f0e640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:19:36.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.011+0000 7f7fb2f0e640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:19:36.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.012+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 >> v1:172.21.15.117:6816/4276038175 conn(0x7f7f9806ea60 legacy=0x7f7f98070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.012+0000 7f7fb2f0e640 10 monclient: shutdown 2024-03-29T11:19:36.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.012+0000 7f7fb2f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:36.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.012+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 >> v1:172.21.15.117:6789/0 conn(0x7f7fac0af900 legacy=0x7f7fac13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:36.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.014+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 shutdown_connections 2024-03-29T11:19:36.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.014+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 >> 172.21.15.117:0/1770945547 conn(0x7f7fac0b35e0 msgr2=0x7f7fac0ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:36.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.017+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 shutdown_connections 2024-03-29T11:19:36.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.017+0000 7f7fb2f0e640 1 -- 172.21.15.117:0/1770945547 wait complete. 2024-03-29T11:19:36.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:37.017+0000 7f7fb2f0e640 1 librados: shutdown 2024-03-29T11:19:37.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:43.820 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.008+0000 7f3336949640 10 monclient: get_monmap_and_config 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.008+0000 7f3336949640 10 monclient: build_initial_monmap 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.008+0000 7f3336949640 10 monclient: monmap: 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:44.009285+0000 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:44.009285+0000 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:43.985 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:43.986 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:43.986 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:43.986 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:43.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.009+0000 7f3336949640 1 Processor -- start 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.009+0000 7f3336949640 1 -- start start 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.009+0000 7f3336949640 10 monclient: init 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient(hunting): picked mon.noname-b con 0x7f333006bef0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient(hunting): picked mon.noname-a con 0x7f333006c2d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient(hunting): picked mon.noname-c con 0x7f333014a010 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333006d070 con 0x7f333006c2d0 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333014a460 con 0x7f333014a010 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33300b0100 con 0x7f333006bef0 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:43.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.010+0000 7f3336949640 10 monclient(hunting): authenticate will time out at 852.319243s 2024-03-29T11:19:43.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.016+0000 7f3335947640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f333006bef0 0x7f3330071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:36933/0 (socket says 172.21.15.117:36933) 2024-03-29T11:19:43.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.016+0000 7f3335947640 1 -- 172.21.15.117:0/1538244557 learned_addr learned my addr 172.21.15.117:0/1538244557 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991576030 0 0) 0x7f333014a460 con 0x7f333014a010 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3318003520 con 0x7f333014a010 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580708676 0 0) 0x7f33300b0100 con 0x7f333006bef0 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _negotiate 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:43.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:43.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:19:43.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.018+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3318003840 con 0x7f333006bef0 2024-03-29T11:19:43.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2823872498 0 0) 0x7f3318003520 con 0x7f333014a010 2024-03-29T11:19:43.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 >> v1:172.21.15.117:6790/0 conn(0x7f333006bef0 legacy=0x7f3330071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 >> v1:172.21.15.117:6789/0 conn(0x7f333006c2d0 legacy=0x7f3330149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:43.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: _finish_hunting 0 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 monclient: found mon.noname-c 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333006cd90 con 0x7f333014a010 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: _finish_auth 0 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: _check_auth_tickets 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3324001670 con 0x7f333014a010 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3334945640 10 monclient: dump: 2024-03-29T11:19:44.000 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:44.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3336949640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.021+0000 7f3336949640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33240017d0 con 0x7f333014a010 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 1 -- 172.21.15.117:0/1538244557 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3324001ba0 con 0x7f333014a010 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3334945640 10 monclient: dump: 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:44.002 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3336949640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:44.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.022+0000 7f3336949640 4 set_mon_vals no callback set 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.023+0000 7f3336949640 10 monclient: shutdown 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.023+0000 7f3336949640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.023+0000 7f3336949640 1 -- 172.21.15.117:0/1538244557 >> v1:172.21.15.173:6789/0 conn(0x7f333014a010 legacy=0x7f3330151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.026+0000 7f3336949640 1 -- 172.21.15.117:0/1538244557 shutdown_connections 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.026+0000 7f3336949640 1 -- 172.21.15.117:0/1538244557 >> 172.21.15.117:0/1538244557 conn(0x7f33300672e0 msgr2=0x7f33300b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:44.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.026+0000 7f3336949640 1 -- 172.21.15.117:0/1538244557 shutdown_connections 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.027+0000 7f3336949640 1 -- 172.21.15.117:0/1538244557 wait complete. 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.027+0000 7f3336949640 10 monclient: build_initial_monmap 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.027+0000 7f3336949640 10 monclient: monmap: 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:44.005 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:44.006 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 librados: starting msgr at 2024-03-29T11:19:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 librados: starting objecter 2024-03-29T11:19:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 Processor -- start 2024-03-29T11:19:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 -- start start 2024-03-29T11:19:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 librados: setting wanted keys 2024-03-29T11:19:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 1 librados: calling monclient init 2024-03-29T11:19:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.028+0000 7f3336949640 10 monclient: init 2024-03-29T11:19:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient(hunting): picked mon.noname-a con 0x7f333006bef0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient(hunting): picked mon.noname-c con 0x7f333006c2d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient(hunting): picked mon.noname-b con 0x7f333014a010 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3330148d30 con 0x7f333006bef0 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3330145b50 con 0x7f333014a010 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3330144d70 con 0x7f333006c2d0 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.029+0000 7f3336949640 10 monclient(hunting): authenticate will time out at 852.338317s 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f3335146640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f333006c2d0 0x7f3330147700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37051/0 (socket says 172.21.15.117:37051) 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f3335146640 1 -- 172.21.15.117:0/3330076984 learned_addr learned my addr 172.21.15.117:0/3330076984 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f332e7fc640 10 client.?.objecter ms_handle_connect 0x7f333006c2d0 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f332e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193025948 0 0) 0x7f3330144d70 con 0x7f333006c2d0 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.034+0000 7f332e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.035+0000 7f332e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.035+0000 7f332e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.036+0000 7f332e7fc640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:19:44.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.036+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331c004eb0 con 0x7f333006c2d0 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.037+0000 7f332e7fc640 10 client.?.objecter ms_handle_connect 0x7f333006bef0 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2016719471 0 0) 0x7f331c004eb0 con 0x7f333006c2d0 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3330144d70 con 0x7f333006c2d0 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3320001d90 con 0x7f333006c2d0 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:44.018 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896389983 0 0) 0x7f3330144d70 con 0x7f333006c2d0 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 >> v1:172.21.15.173:6789/0 conn(0x7f333014a010 legacy=0x7f3330147e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:19:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 >> v1:172.21.15.117:6789/0 conn(0x7f333006bef0 legacy=0x7f3330145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 monclient: found mon.c 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330144f00 con 0x7f333006c2d0 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.038+0000 7f332e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 10 monclient: _renew_subs 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3330146e00 con 0x7f333006c2d0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 10 monclient: _renew_subs 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3330135190 con 0x7f333006c2d0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f3336949640 1 librados: init done 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3320001ef0 con 0x7f333006c2d0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3320003de0 con 0x7f333006c2d0 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332ffff640 4 set_mon_vals no callback set 2024-03-29T11:19:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.039+0000 7f332e7fc640 10 monclient: dump: 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:44.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.041+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f3320003f40 con 0x7f333006c2d0 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.041+0000 7f3336949640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.041+0000 7f3336949640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.041+0000 7f3336949640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.041+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310005080 con 0x7f333006c2d0 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f3335947640 10 monclient: get_auth_request con 0x7f331c06dc40 auth_method 0 2024-03-29T11:19:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f3320084f40 con 0x7f333006c2d0 2024-03-29T11:19:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 10 client.4259.objecter ms_dispatch 0x7f3330000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:19:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.043+0000 7f332e7fc640 10 client.4259.objecter ms_handle_connect 0x7f331c06dc40 2024-03-29T11:19:44.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.054+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3320050880 con 0x7f333006c2d0 2024-03-29T11:19:44.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.054+0000 7f332e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:44.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.054+0000 7f332e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:44.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.382+0000 7f3336949640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.382+0000 7f3336949640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.382+0000 7f3336949640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:44.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.382+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3310005330 con 0x7f333006c2d0 2024-03-29T11:19:44.363 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:44.363 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:44.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.386+0000 7f332e7fc640 1 -- 172.21.15.117:0/3330076984 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f33200527b0 con 0x7f333006c2d0 2024-03-29T11:19:44.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.386+0000 7f332e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:44.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.386+0000 7f332e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:44.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 10 librados: watch_flush enter 2024-03-29T11:19:44.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 10 librados: watch_flush exit 2024-03-29T11:19:44.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:19:44.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 >> v1:172.21.15.117:6816/4276038175 conn(0x7f331c06dc40 legacy=0x7f331c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 10 monclient: shutdown 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.389+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 >> v1:172.21.15.117:6790/0 conn(0x7f333006c2d0 legacy=0x7f3330147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.390+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 shutdown_connections 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.390+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 >> 172.21.15.117:0/3330076984 conn(0x7f33300672e0 msgr2=0x7f33300b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.390+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 shutdown_connections 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.390+0000 7f3336949640 1 -- 172.21.15.117:0/3330076984 wait complete. 2024-03-29T11:19:44.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:44.390+0000 7f3336949640 1 librados: shutdown 2024-03-29T11:19:44.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:51.192 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:51.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.371+0000 7f27a31e8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.371+0000 7f27a31e8640 10 monclient: build_initial_monmap 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.372+0000 7f27a31e8640 10 monclient: monmap: 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:51.373060+0000 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:51.373060+0000 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:51.350 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.374+0000 7f27a31e8640 1 Processor -- start 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.374+0000 7f27a31e8640 1 -- start start 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.374+0000 7f27a31e8640 10 monclient: init 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-b con 0x7f279c0a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-a con 0x7f279c13ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-c con 0x7f279c0af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0afd10 con 0x7f279c13ee90 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0af6f0 con 0x7f279c0af900 2024-03-29T11:19:51.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0aac80 con 0x7f279c0a9ac0 2024-03-29T11:19:51.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:51.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.375+0000 7f27a31e8640 10 monclient(hunting): authenticate will time out at 859.684599s 2024-03-29T11:19:51.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.378+0000 7f279bfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f279c13ee90 0x7f279c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53343/0 (socket says 172.21.15.117:53343) 2024-03-29T11:19:51.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.378+0000 7f279bfff640 1 -- 172.21.15.117:0/2637858256 learned_addr learned my addr 172.21.15.117:0/2637858256 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30412726 0 0) 0x7f279c0afd10 con 0x7f279c13ee90 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:19:51.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.379+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2784003520 con 0x7f279c13ee90 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.380+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694490412 0 0) 0x7f2784003520 con 0x7f279c13ee90 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.380+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 >> v1:172.21.15.117:6790/0 conn(0x7f279c0a9ac0 legacy=0x7f279c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 >> v1:172.21.15.173:6789/0 conn(0x7f279c0af900 legacy=0x7f279c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c0aaf60 con 0x7f279c13ee90 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f278c001690 con 0x7f279c13ee90 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f279b7fe640 10 monclient: dump: 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:51.359 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f27a31e8640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:19:51.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.381+0000 7f27a31e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:51.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.384+0000 7f279b7fe640 1 -- 172.21.15.117:0/2637858256 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f278c001860 con 0x7f279c13ee90 2024-03-29T11:19:51.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.384+0000 7f279b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:51.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.384+0000 7f27a31e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:51.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.384+0000 7f27a31e8640 4 set_mon_vals no callback set 2024-03-29T11:19:51.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.385+0000 7f27a31e8640 10 monclient: shutdown 2024-03-29T11:19:51.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.385+0000 7f27a31e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:51.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.385+0000 7f27a31e8640 1 -- 172.21.15.117:0/2637858256 >> v1:172.21.15.117:6789/0 conn(0x7f279c13ee90 legacy=0x7f279c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.386+0000 7f27a31e8640 1 -- 172.21.15.117:0/2637858256 shutdown_connections 2024-03-29T11:19:51.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.386+0000 7f27a31e8640 1 -- 172.21.15.117:0/2637858256 >> 172.21.15.117:0/2637858256 conn(0x7f279c0b35e0 msgr2=0x7f279c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:51.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.387+0000 7f27a31e8640 1 -- 172.21.15.117:0/2637858256 shutdown_connections 2024-03-29T11:19:51.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.387+0000 7f27a31e8640 1 -- 172.21.15.117:0/2637858256 wait complete. 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.388+0000 7f27a31e8640 10 monclient: build_initial_monmap 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.388+0000 7f27a31e8640 10 monclient: monmap: 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:51.366 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.388+0000 7f27a31e8640 1 librados: starting msgr at 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.388+0000 7f27a31e8640 1 librados: starting objecter 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.388+0000 7f27a31e8640 1 Processor -- start 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.389+0000 7f27a31e8640 1 -- start start 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.389+0000 7f27a31e8640 1 librados: setting wanted keys 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.389+0000 7f27a31e8640 1 librados: calling monclient init 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.389+0000 7f27a31e8640 10 monclient: init 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.389+0000 7f27a31e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:51.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-c con 0x7f279c0a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-b con 0x7f279c0af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient(hunting): picked mon.noname-a con 0x7f279c13ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c070a50 con 0x7f279c13ee90 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c072020 con 0x7f279c0af900 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c072290 con 0x7f279c0a9ac0 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:51.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.390+0000 7f27a31e8640 10 monclient(hunting): authenticate will time out at 859.699103s 2024-03-29T11:19:51.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.394+0000 7f27a1785640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f279c13ee90 0x7f279c2007b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53417/0 (socket says 172.21.15.117:53417) 2024-03-29T11:19:51.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.394+0000 7f27a1785640 1 -- 172.21.15.117:0/1924309649 learned_addr learned my addr 172.21.15.117:0/1924309649 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 client.?.objecter ms_handle_connect 0x7f279c13ee90 2024-03-29T11:19:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 client.?.objecter ms_handle_connect 0x7f279c0a9ac0 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420695638 0 0) 0x7f279c070a50 con 0x7f279c13ee90 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.396+0000 7f27997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2788004eb0 con 0x7f279c13ee90 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477887618 0 0) 0x7f279c072290 con 0x7f279c0a9ac0 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c070a50 con 0x7f279c0a9ac0 2024-03-29T11:19:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 client.?.objecter ms_handle_connect 0x7f279c0af900 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3784176631 0 0) 0x7f279c070a50 con 0x7f279c0a9ac0 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f279c072290 con 0x7f279c0a9ac0 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f27900017a0 con 0x7f279c0a9ac0 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): dump: 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:51.379 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680309900 0 0) 0x7f2788004eb0 con 0x7f279c13ee90 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f279c070a50 con 0x7f279c13ee90 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2794001d10 con 0x7f279c13ee90 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.397+0000 7f27997fa640 10 monclient(hunting): dump: 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:51.380 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.398+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342400965 0 0) 0x7f279c072020 con 0x7f279c0af900 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2788004eb0 con 0x7f279c0af900 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1724275313 0 0) 0x7f279c072290 con 0x7f279c0a9ac0 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 >> v1:172.21.15.173:6789/0 conn(0x7f279c0af900 legacy=0x7f279c14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 >> v1:172.21.15.117:6789/0 conn(0x7f279c13ee90 legacy=0x7f279c2007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.399+0000 7f27997fa640 1 monclient: found mon.c 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c06eeb0 con 0x7f279c0a9ac0 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:51.381 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 10 monclient: _renew_subs 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c0700f0 con 0x7f279c0a9ac0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 10 monclient: _renew_subs 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c070610 con 0x7f279c0a9ac0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27a31e8640 1 librados: init done 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2790001c90 con 0x7f279c0a9ac0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2790003790 con 0x7f279c0a9ac0 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f279affd640 4 set_mon_vals no callback set 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.400+0000 7f27997fa640 10 monclient: dump: 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:51.383 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.401+0000 7f27a31e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.401+0000 7f27a31e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.401+0000 7f27a31e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.401+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277c005080 con 0x7f279c0a9ac0 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.405+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f27900038f0 con 0x7f279c0a9ac0 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.406+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f2790083720 con 0x7f279c0a9ac0 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.406+0000 7f27997fa640 10 client.4265.objecter ms_dispatch 0x7f279c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.406+0000 7f27997fa640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.406+0000 7f27997fa640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.406+0000 7f27997fa640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:19:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.408+0000 7f279bfff640 10 monclient: get_auth_request con 0x7f278806dfe0 auth_method 0 2024-03-29T11:19:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.409+0000 7f27997fa640 10 client.4265.objecter ms_handle_connect 0x7f278806dfe0 2024-03-29T11:19:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.416+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f279004f000 con 0x7f279c0a9ac0 2024-03-29T11:19:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.416+0000 7f27997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.416+0000 7f27997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:51.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.735+0000 7f27a31e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.735+0000 7f27a31e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.735+0000 7f27a31e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:51.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.735+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f277c005330 con 0x7f279c0a9ac0 2024-03-29T11:19:51.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.739+0000 7f27997fa640 1 -- 172.21.15.117:0/1924309649 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f2790050f30 con 0x7f279c0a9ac0 2024-03-29T11:19:51.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.739+0000 7f27997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:51.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.739+0000 7f27997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:51.716 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:51.717 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:51.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.743+0000 7f27a31e8640 10 librados: watch_flush enter 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.745+0000 7f27a31e8640 10 librados: watch_flush exit 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.745+0000 7f27a31e8640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.745+0000 7f27a31e8640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 >> v1:172.21.15.117:6816/4276038175 conn(0x7f278806dfe0 legacy=0x7f27880704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 10 monclient: shutdown 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:51.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 >> v1:172.21.15.117:6790/0 conn(0x7f279c0a9ac0 legacy=0x7f279c06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:51.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 shutdown_connections 2024-03-29T11:19:51.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 >> 172.21.15.117:0/1924309649 conn(0x7f279c0b35e0 msgr2=0x7f279c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:51.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 shutdown_connections 2024-03-29T11:19:51.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.746+0000 7f27a31e8640 1 -- 172.21.15.117:0/1924309649 wait complete. 2024-03-29T11:19:51.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:51.747+0000 7f27a31e8640 1 librados: shutdown 2024-03-29T11:19:51.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:58.546 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.723+0000 7fdbbc5bc640 10 monclient: get_monmap_and_config 2024-03-29T11:19:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.723+0000 7fdbbc5bc640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.723+0000 7fdbbc5bc640 10 monclient: monmap: 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:19:58.725010+0000 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:19:58.725010+0000 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:19:58.701 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.724+0000 7fdbbc5bc640 1 Processor -- start 2024-03-29T11:19:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.724+0000 7fdbbc5bc640 1 -- start start 2024-03-29T11:19:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.724+0000 7fdbbc5bc640 10 monclient: init 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb40b4790 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb40aea90 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb40b39f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb40b0c60 con 0x7fdbb40b39f0 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb40af020 con 0x7fdbb40b4790 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb413fbd0 con 0x7fdbb40aea90 2024-03-29T11:19:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.725+0000 7fdbbc5bc640 10 monclient(hunting): authenticate will time out at 867.034966s 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbba358640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fdbb40b4790 0x7fdbb40b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53291/0 (socket says 172.21.15.117:53291) 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbba358640 1 -- 172.21.15.117:0/1896297648 learned_addr learned my addr 172.21.15.117:0/1896297648 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462297670 0 0) 0x7fdbb40af020 con 0x7fdbb40b4790 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.729+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba0003520 con 0x7fdbb40b4790 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.730+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 131033903 0 0) 0x7fdba0003520 con 0x7fdbb40b4790 2024-03-29T11:19:58.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.730+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 >> v1:172.21.15.117:6790/0 conn(0x7fdbb40aea90 legacy=0x7fdbb40af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:58.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.735+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 >> v1:172.21.15.117:6789/0 conn(0x7fdbb40b39f0 legacy=0x7fdbb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:58.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.735+0000 7fdbb9356640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 1 monclient: found mon.noname-c 2024-03-29T11:19:58.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb413fd60 con 0x7fdbb40b4790 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdbac001a10 con 0x7fdbb40b4790 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.736+0000 7fdbb9356640 10 monclient: dump: 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:58.718 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbbc5bc640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbbc5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac001be0 con 0x7fdbb40b4790 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbbc5bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbbc5bc640 4 set_mon_vals no callback set 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 1 -- 172.21.15.117:0/1896297648 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdbac001ff0 con 0x7fdbb40b4790 2024-03-29T11:19:58.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.740+0000 7fdbb9356640 10 monclient: dump: 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 10 monclient: shutdown 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/1896297648 >> v1:172.21.15.173:6789/0 conn(0x7fdbb40b4790 legacy=0x7fdbb40b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/1896297648 shutdown_connections 2024-03-29T11:19:58.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/1896297648 >> 172.21.15.117:0/1896297648 conn(0x7fdbb4066a00 msgr2=0x7fdbb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:58.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/1896297648 shutdown_connections 2024-03-29T11:19:58.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.741+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/1896297648 wait complete. 2024-03-29T11:19:58.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.742+0000 7fdbbc5bc640 10 monclient: build_initial_monmap 2024-03-29T11:19:58.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.742+0000 7fdbbc5bc640 10 monclient: monmap: 2024-03-29T11:19:58.722 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 librados: starting msgr at 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 librados: starting objecter 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 Processor -- start 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 -- start start 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 librados: setting wanted keys 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 1 librados: calling monclient init 2024-03-29T11:19:58.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.743+0000 7fdbbc5bc640 10 monclient: init 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb40b39f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-a con 0x7fdbb40b4790 addr v1:172.21.15.117:6789/0 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient(hunting): picked mon.noname-b con 0x7fdbb414e0e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb41d5290 con 0x7fdbb40b4790 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb413ff70 con 0x7fdbb414e0e0 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb41e8450 con 0x7fdbb40b39f0 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:58.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.744+0000 7fdbbc5bc640 10 monclient(hunting): authenticate will time out at 867.053354s 2024-03-29T11:19:58.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.748+0000 7fdbbab59640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fdbb414e0e0 0x7fdbb41e7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53371/0 (socket says 172.21.15.117:53371) 2024-03-29T11:19:58.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.748+0000 7fdbbab59640 1 -- 172.21.15.117:0/11782072 learned_addr learned my addr 172.21.15.117:0/11782072 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.751+0000 7fdbaaffd640 10 client.?.objecter ms_handle_connect 0x7fdbb414e0e0 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 client.?.objecter ms_handle_connect 0x7fdbb40b39f0 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943136393 0 0) 0x7fdbb41e8450 con 0x7fdbb40b39f0 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb98004eb0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 client.?.objecter ms_handle_connect 0x7fdbb40b4790 2024-03-29T11:19:58.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.752+0000 7fdbaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:58.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481044017 0 0) 0x7fdbb41d5290 con 0x7fdbb40b4790 2024-03-29T11:19:58.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:58.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:58.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:58.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:19:58.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.753+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb41e8450 con 0x7fdbb40b4790 2024-03-29T11:19:58.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3009780014 0 0) 0x7fdb98004eb0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbb41d5290 con 0x7fdbb40b39f0 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdbac001ef0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient(hunting): dump: 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:58.733 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2135070284 0 0) 0x7fdbb41d5290 con 0x7fdbb40b39f0 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 >> v1:172.21.15.173:6789/0 conn(0x7fdbb414e0e0 legacy=0x7fdbb41e7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 >> v1:172.21.15.117:6789/0 conn(0x7fdbb40b4790 legacy=0x7fdbb414d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 monclient: found mon.c 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb41e85e0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 10 monclient: _renew_subs 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb41414c0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 10 monclient: _renew_subs 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb41f0ca0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.756+0000 7fdbbc5bc640 1 librados: init done 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbac004e10 con 0x7fdbb40b39f0 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:58.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdbac005120 con 0x7fdbb40b39f0 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbb8b55640 4 set_mon_vals no callback set 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.757+0000 7fdbaaffd640 10 monclient: dump: 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:19:58.736 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:19:58.737 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:19:58.737 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:19:58.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.760+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fdbac01b0f0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.760+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fdbac084e10 con 0x7fdbb40b39f0 2024-03-29T11:19:58.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbaaffd640 10 client.4271.objecter ms_dispatch 0x7fdbb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:58.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbaaffd640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:58.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbaaffd640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:58.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbaaffd640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:19:58.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbb9b57640 10 monclient: get_auth_request con 0x7fdb9806de40 auth_method 0 2024-03-29T11:19:58.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.761+0000 7fdbaaffd640 10 client.4271.objecter ms_handle_connect 0x7fdb9806de40 2024-03-29T11:19:58.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.763+0000 7fdbbc5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.764+0000 7fdbbc5bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.764+0000 7fdbbc5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:58.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.764+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c005080 con 0x7fdbb40b39f0 2024-03-29T11:19:58.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.774+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbac0506f0 con 0x7fdbb40b39f0 2024-03-29T11:19:58.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.774+0000 7fdbaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:58.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:58.774+0000 7fdbaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:59.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.101+0000 7fdbbc5bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.101+0000 7fdbbc5bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.101+0000 7fdbbc5bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:19:59.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.101+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb9c005330 con 0x7fdbb40b39f0 2024-03-29T11:19:59.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.104+0000 7fdbaaffd640 1 -- 172.21.15.117:0/11782072 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fdbac052620 con 0x7fdbb40b39f0 2024-03-29T11:19:59.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.104+0000 7fdbaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:59.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.104+0000 7fdbaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:59.081 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:19:59.081 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.107+0000 7fdbbc5bc640 10 librados: watch_flush enter 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.107+0000 7fdbbc5bc640 10 librados: watch_flush exit 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.107+0000 7fdbbc5bc640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.107+0000 7fdbbc5bc640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 >> v1:172.21.15.117:6816/4276038175 conn(0x7fdb9806de40 legacy=0x7fdb98070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 10 monclient: shutdown 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 >> v1:172.21.15.117:6790/0 conn(0x7fdbb40b39f0 legacy=0x7fdbb4140700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 shutdown_connections 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 >> 172.21.15.117:0/11782072 conn(0x7fdbb4066a00 msgr2=0x7fdbb40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 shutdown_connections 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 -- 172.21.15.117:0/11782072 wait complete. 2024-03-29T11:19:59.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:19:59.108+0000 7fdbbc5bc640 1 librados: shutdown 2024-03-29T11:19:59.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:05.908 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:06.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.078+0000 7f9f76d6d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.078+0000 7f9f76d6d640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.078+0000 7f9f76d6d640 10 monclient: monmap: 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:06.080028+0000 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:06.080028+0000 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:06.056 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:06.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.081+0000 7f9f76d6d640 1 Processor -- start 2024-03-29T11:20:06.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.081+0000 7f9f76d6d640 1 -- start start 2024-03-29T11:20:06.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.081+0000 7f9f76d6d640 10 monclient: init 2024-03-29T11:20:06.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f701457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7013f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f9f700a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700aaef0 con 0x7f9f700a9d70 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f70145bd0 con 0x7f9f701457f0 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f70145eb0 con 0x7f9f7013f650 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.082+0000 7f9f76d6d640 10 monclient(hunting): authenticate will time out at 874.391400s 2024-03-29T11:20:06.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.087+0000 7f9f7530a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f9f700a9d70 0x7f9f7014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:49227/0 (socket says 172.21.15.117:49227) 2024-03-29T11:20:06.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.087+0000 7f9f7530a640 1 -- 172.21.15.117:0/3024370389 learned_addr learned my addr 172.21.15.117:0/3024370389 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493952466 0 0) 0x7f9f70145eb0 con 0x7f9f7013f650 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:06.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58003520 con 0x7f9f7013f650 2024-03-29T11:20:06.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626584875 0 0) 0x7f9f700aaef0 con 0x7f9f700a9d70 2024-03-29T11:20:06.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.088+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f70145eb0 con 0x7f9f700a9d70 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063053935 0 0) 0x7f9f70145eb0 con 0x7f9f700a9d70 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 >> v1:172.21.15.117:6790/0 conn(0x7f9f7013f650 legacy=0x7f9f700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 >> v1:172.21.15.173:6789/0 conn(0x7f9f701457f0 legacy=0x7f9f7013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70146340 con 0x7f9f700a9d70 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.089+0000 7f9f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f76d6d640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f76d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:06.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f640015a0 con 0x7f9f700a9d70 2024-03-29T11:20:06.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 10 monclient: dump: 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:06.070 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f76d6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 1 -- 172.21.15.117:0/3024370389 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f640017c0 con 0x7f9f700a9d70 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f76d6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.090+0000 7f9f76d6d640 4 set_mon_vals no callback set 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.091+0000 7f9f76d6d640 10 monclient: shutdown 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.091+0000 7f9f76d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.091+0000 7f9f76d6d640 1 -- 172.21.15.117:0/3024370389 >> v1:172.21.15.117:6789/0 conn(0x7f9f700a9d70 legacy=0x7f9f7014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.091+0000 7f9f76d6d640 1 -- 172.21.15.117:0/3024370389 shutdown_connections 2024-03-29T11:20:06.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.092+0000 7f9f76d6d640 1 -- 172.21.15.117:0/3024370389 >> 172.21.15.117:0/3024370389 conn(0x7f9f70066a00 msgr2=0x7f9f70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.094+0000 7f9f76d6d640 1 -- 172.21.15.117:0/3024370389 shutdown_connections 2024-03-29T11:20:06.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.094+0000 7f9f76d6d640 1 -- 172.21.15.117:0/3024370389 wait complete. 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 10 monclient: build_initial_monmap 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 10 monclient: monmap: 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:06.074 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 librados: starting msgr at 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 librados: starting objecter 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 Processor -- start 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 -- start start 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 librados: setting wanted keys 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 1 librados: calling monclient init 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.095+0000 7f9f76d6d640 10 monclient: init 2024-03-29T11:20:06.075 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.096+0000 7f9f76d6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:06.076 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.096+0000 7f9f76d6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-a con 0x7f9f700a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7013f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 10 monclient(hunting): picked mon.noname-b con 0x7f9f70142a60 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7006dfc0 con 0x7f9f700a9d70 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7006f210 con 0x7f9f70142a60 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f70141cc0 con 0x7f9f7013f650 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.098+0000 7f9f76d6d640 10 monclient(hunting): authenticate will time out at 874.407370s 2024-03-29T11:20:06.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.100+0000 7f9f74b09640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f9f700a9d70 0x7f9f7013fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:49351/0 (socket says 172.21.15.117:49351) 2024-03-29T11:20:06.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.100+0000 7f9f74b09640 1 -- 172.21.15.117:0/4188982555 learned_addr learned my addr 172.21.15.117:0/4188982555 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:06.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f9f700a9d70 2024-03-29T11:20:06.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227566434 0 0) 0x7f9f7006dfc0 con 0x7f9f700a9d70 2024-03-29T11:20:06.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5c004eb0 con 0x7f9f700a9d70 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f9f7013f650 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.101+0000 7f9f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257180883 0 0) 0x7f9f70141cc0 con 0x7f9f7013f650 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7006dfc0 con 0x7f9f7013f650 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3462186496 0 0) 0x7f9f5c004eb0 con 0x7f9f700a9d70 2024-03-29T11:20:06.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f70141cc0 con 0x7f9f700a9d70 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f9f70142a60 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f68001bf0 con 0x7f9f700a9d70 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.102+0000 7f9f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): dump: 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:06.085 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151583100 0 0) 0x7f9f7006f210 con 0x7f9f70142a60 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5c004eb0 con 0x7f9f70142a60 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2531034309 0 0) 0x7f9f70141cc0 con 0x7f9f700a9d70 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 >> v1:172.21.15.117:6790/0 conn(0x7f9f7013f650 legacy=0x7f9f70140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 >> v1:172.21.15.173:6789/0 conn(0x7f9f70142a60 legacy=0x7f9f701eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 monclient: found mon.a 2024-03-29T11:20:06.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f70137ef0 con 0x7f9f700a9d70 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.103+0000 7f9f6d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f6d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f68001d50 con 0x7f9f700a9d70 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 10 monclient: _renew_subs 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7013b360 con 0x7f9f700a9d70 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 10 monclient: _renew_subs 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f701f0320 con 0x7f9f700a9d70 2024-03-29T11:20:06.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.104+0000 7f9f76d6d640 1 librados: init done 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f68004db0 con 0x7f9f700a9d70 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6d7fa640 10 monclient: dump: 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:06.088 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:06.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.105+0000 7f9f6effd640 4 set_mon_vals no callback set 2024-03-29T11:20:06.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.107+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f9f6801ad80 con 0x7f9f700a9d70 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6ffff640 10 monclient: get_auth_request con 0x7f9f5c06dea0 auth_method 0 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6d7fa640 10 client.4278.objecter ms_handle_connect 0x7f9f5c06dea0 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f9f68084d10 con 0x7f9f700a9d70 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6d7fa640 10 client.4278.objecter ms_dispatch 0x7f9f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6d7fa640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.108+0000 7f9f6d7fa640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.109+0000 7f9f6d7fa640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.109+0000 7f9f76d6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.109+0000 7f9f76d6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.109+0000 7f9f76d6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.091 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.109+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f50005080 con 0x7f9f700a9d70 2024-03-29T11:20:06.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.119+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f680505f0 con 0x7f9f700a9d70 2024-03-29T11:20:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.119+0000 7f9f6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:06.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.119+0000 7f9f6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:06.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.450+0000 7f9f76d6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.450+0000 7f9f76d6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.450+0000 7f9f76d6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:06.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.450+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f50005330 con 0x7f9f700a9d70 2024-03-29T11:20:06.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.455+0000 7f9f6d7fa640 1 -- 172.21.15.117:0/4188982555 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f9f68052520 con 0x7f9f700a9d70 2024-03-29T11:20:06.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.455+0000 7f9f6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:06.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.455+0000 7f9f6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:06.432 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:06.432 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:06.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.458+0000 7f9f76d6d640 10 librados: watch_flush enter 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 10 librados: watch_flush exit 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 >> v1:172.21.15.117:6816/4276038175 conn(0x7f9f5c06dea0 legacy=0x7f9f5c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 10 monclient: shutdown 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:06.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.460+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 >> v1:172.21.15.117:6789/0 conn(0x7f9f700a9d70 legacy=0x7f9f7013fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:06.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.461+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 shutdown_connections 2024-03-29T11:20:06.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.461+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 >> 172.21.15.117:0/4188982555 conn(0x7f9f70066a00 msgr2=0x7f9f700b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:06.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.461+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 shutdown_connections 2024-03-29T11:20:06.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.463+0000 7f9f76d6d640 1 -- 172.21.15.117:0/4188982555 wait complete. 2024-03-29T11:20:06.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:06.465+0000 7f9f76d6d640 1 librados: shutdown 2024-03-29T11:20:06.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:13.270 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.446+0000 7f7f2cbd7640 10 monclient: get_monmap_and_config 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.446+0000 7f7f2cbd7640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.446+0000 7f7f2cbd7640 10 monclient: monmap: 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:13.448011+0000 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:13.448011+0000 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.424 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:13.425 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:13.425 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:13.425 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.448+0000 7f7f2cbd7640 1 Processor -- start 2024-03-29T11:20:13.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.448+0000 7f7f2cbd7640 1 -- start start 2024-03-29T11:20:13.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.448+0000 7f7f2cbd7640 10 monclient: init 2024-03-29T11:20:13.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.450+0000 7f7f2cbd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.450+0000 7f7f2cbd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:13.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.450+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f7f280b0770 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.450+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f7f2814a700 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f7f280ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280aba70 con 0x7f7f280b0770 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280aecb0 con 0x7f7f2814a700 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280af150 con 0x7f7f280ac5a0 2024-03-29T11:20:13.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.452+0000 7f7f2cbd7640 10 monclient(hunting): authenticate will time out at 881.761553s 2024-03-29T11:20:13.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.458+0000 7f7f26d9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7f280ac5a0 0x7f7f28151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:36405/0 (socket says 172.21.15.117:36405) 2024-03-29T11:20:13.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.458+0000 7f7f26d9d640 1 -- 172.21.15.117:0/3185736570 learned_addr learned my addr 172.21.15.117:0/3185736570 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:13.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807406992 0 0) 0x7f7f280af150 con 0x7f7f280ac5a0 2024-03-29T11:20:13.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:20:13.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10003520 con 0x7f7f280ac5a0 2024-03-29T11:20:13.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059540114 0 0) 0x7f7f10003520 con 0x7f7f280ac5a0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 >> v1:172.21.15.173:6789/0 conn(0x7f7f2814a700 legacy=0x7f7f280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.459+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 >> v1:172.21.15.117:6789/0 conn(0x7f7f280b0770 legacy=0x7f7f28149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 1 monclient: found mon.noname-b 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280ac9d0 con 0x7f7f280ac5a0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7f1c001690 con 0x7f7f280ac5a0 2024-03-29T11:20:13.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: dump: 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f1c001860 con 0x7f7f280ac5a0 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 1 -- 172.21.15.117:0/3185736570 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7f1c001c70 con 0x7f7f280ac5a0 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.460+0000 7f7f2559a640 10 monclient: dump: 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:13.440 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.461+0000 7f7f2cbd7640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.461+0000 7f7f2cbd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.461+0000 7f7f2cbd7640 4 set_mon_vals no callback set 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.462+0000 7f7f2cbd7640 10 monclient: shutdown 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.462+0000 7f7f2cbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.462+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/3185736570 >> v1:172.21.15.117:6790/0 conn(0x7f7f280ac5a0 legacy=0x7f7f28151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.463+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/3185736570 shutdown_connections 2024-03-29T11:20:13.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.463+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/3185736570 >> 172.21.15.117:0/3185736570 conn(0x7f7f2806b660 msgr2=0x7f7f2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.467+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/3185736570 shutdown_connections 2024-03-29T11:20:13.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.467+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/3185736570 wait complete. 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.468+0000 7f7f2cbd7640 10 monclient: build_initial_monmap 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.468+0000 7f7f2cbd7640 10 monclient: monmap: 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:13.445 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 librados: starting msgr at 2024-03-29T11:20:13.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 librados: starting objecter 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 Processor -- start 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 -- start start 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 librados: setting wanted keys 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 1 librados: calling monclient init 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.469+0000 7f7f2cbd7640 10 monclient: init 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-c con 0x7f7f280ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-b con 0x7f7f280b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2814a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f281278c0 con 0x7f7f2814a700 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f28127b10 con 0x7f7f280b0770 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f28147260 con 0x7f7f280ac5a0 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:13.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.470+0000 7f7f2cbd7640 10 monclient(hunting): authenticate will time out at 881.779797s 2024-03-29T11:20:13.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.475+0000 7f7f26d9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7f2814a700 0x7f7f281dba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:60621/0 (socket says 172.21.15.117:60621) 2024-03-29T11:20:13.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.475+0000 7f7f26d9d640 1 -- 172.21.15.117:0/1420302829 learned_addr learned my addr 172.21.15.117:0/1420302829 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.475+0000 7f7f16ffd640 10 client.?.objecter ms_handle_connect 0x7f7f2814a700 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.475+0000 7f7f16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328251595 0 0) 0x7f7f281278c0 con 0x7f7f2814a700 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f08004eb0 con 0x7f7f2814a700 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 client.?.objecter ms_handle_connect 0x7f7f280b0770 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896264380 0 0) 0x7f7f28127b10 con 0x7f7f280b0770 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:13.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f281278c0 con 0x7f7f280b0770 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577589631 0 0) 0x7f7f08004eb0 con 0x7f7f2814a700 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f28127b10 con 0x7f7f2814a700 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7f1c001b70 con 0x7f7f2814a700 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1049948425 0 0) 0x7f7f28127b10 con 0x7f7f2814a700 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 >> v1:172.21.15.117:6790/0 conn(0x7f7f280ac5a0 legacy=0x7f7f28146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 >> v1:172.21.15.173:6789/0 conn(0x7f7f280b0770 legacy=0x7f7f281db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 1 monclient: found mon.a 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.476+0000 7f7f16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f281473f0 con 0x7f7f2814a700 2024-03-29T11:20:13.459 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 10 monclient: _renew_subs 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f28128030 con 0x7f7f2814a700 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 10 monclient: _renew_subs 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f28128560 con 0x7f7f2814a700 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f2cbd7640 1 librados: init done 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f1c004bf0 con 0x7f7f2814a700 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7f1c004f00 con 0x7f7f2814a700 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:13.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.477+0000 7f7f16ffd640 10 monclient: dump: 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.478+0000 7f7f17fff640 4 set_mon_vals no callback set 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.479+0000 7f7f2cbd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.479+0000 7f7f2cbd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.479+0000 7f7f2cbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:13.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.479+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c005080 con 0x7f7f2814a700 2024-03-29T11:20:13.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7f1c01aed0 con 0x7f7f2814a700 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f7f1c084b70 con 0x7f7f2814a700 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 10 client.4281.objecter ms_dispatch 0x7f7f28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.482+0000 7f7f16ffd640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.483+0000 7f7f2659c640 10 monclient: get_auth_request con 0x7f7f0806dd00 auth_method 0 2024-03-29T11:20:13.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.484+0000 7f7f16ffd640 10 client.4281.objecter ms_handle_connect 0x7f7f0806dd00 2024-03-29T11:20:13.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.491+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f1c050450 con 0x7f7f2814a700 2024-03-29T11:20:13.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.491+0000 7f7f16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:13.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.491+0000 7f7f16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:13.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.821+0000 7f7f2cbd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.821+0000 7f7f2cbd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.821+0000 7f7f2cbd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:13.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.821+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f0c005330 con 0x7f7f2814a700 2024-03-29T11:20:13.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.822+0000 7f7f16ffd640 1 -- 172.21.15.117:0/1420302829 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f7f1c052380 con 0x7f7f2814a700 2024-03-29T11:20:13.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.822+0000 7f7f16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:13.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.822+0000 7f7f16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:13.800 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:13.800 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:13.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 10 librados: watch_flush enter 2024-03-29T11:20:13.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 10 librados: watch_flush exit 2024-03-29T11:20:13.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:20:13.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:20:13.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 >> v1:172.21.15.117:6816/4276038175 conn(0x7f7f0806dd00 legacy=0x7f7f080701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 10 monclient: shutdown 2024-03-29T11:20:13.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:13.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.826+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 >> v1:172.21.15.117:6789/0 conn(0x7f7f2814a700 legacy=0x7f7f281dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:13.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.830+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 shutdown_connections 2024-03-29T11:20:13.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.830+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 >> 172.21.15.117:0/1420302829 conn(0x7f7f2806b660 msgr2=0x7f7f2806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:13.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.832+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 shutdown_connections 2024-03-29T11:20:13.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.835+0000 7f7f2cbd7640 1 -- 172.21.15.117:0/1420302829 wait complete. 2024-03-29T11:20:13.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:13.835+0000 7f7f2cbd7640 1 librados: shutdown 2024-03-29T11:20:13.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:20.637 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.818+0000 7fca12445640 10 monclient: get_monmap_and_config 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.818+0000 7fca12445640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.818+0000 7fca12445640 10 monclient: monmap: 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:20.819946+0000 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:20.819946+0000 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.796 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:20.797 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:20.797 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:20.797 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.820+0000 7fca12445640 1 Processor -- start 2024-03-29T11:20:20.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.820+0000 7fca12445640 1 -- start start 2024-03-29T11:20:20.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.820+0000 7fca12445640 10 monclient: init 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0b0500 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c0b08e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0aeca0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c140620 con 0x7fca0c0b08e0 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c13daf0 con 0x7fca0c0aeca0 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0ae180 con 0x7fca0c0b0500 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca12445640 10 monclient(hunting): authenticate will time out at 889.130564s 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca0b7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fca0c0b08e0 0x7fca0c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53156/0 (socket says 172.21.15.117:53156) 2024-03-29T11:20:20.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.821+0000 7fca0b7fe640 1 -- 172.21.15.117:0/410317262 learned_addr learned my addr 172.21.15.117:0/410317262 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282216212 0 0) 0x7fca0c0ae180 con 0x7fca0c0b0500 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f4003520 con 0x7fca0c0b0500 2024-03-29T11:20:20.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324090139 0 0) 0x7fca0c140620 con 0x7fca0c0b08e0 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c0ae180 con 0x7fca0c0b08e0 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439087705 0 0) 0x7fca0c13daf0 con 0x7fca0c0aeca0 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.822+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c140620 con 0x7fca0c0aeca0 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2317876553 0 0) 0x7fc9f4003520 con 0x7fca0c0b0500 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 >> v1:172.21.15.173:6789/0 conn(0x7fca0c0aeca0 legacy=0x7fca0c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 >> v1:172.21.15.117:6789/0 conn(0x7fca0c0b08e0 legacy=0x7fca0c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 monclient: found mon.noname-b 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0accf0 con 0x7fca0c0b0500 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9fc001690 con 0x7fca0c0b0500 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca0affd640 10 monclient: dump: 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:20.804 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca12445640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.823+0000 7fca12445640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9fc0017f0 con 0x7fca0c0b0500 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 1 -- 172.21.15.117:0/410317262 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9fc001c00 con 0x7fca0c0b0500 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca0affd640 10 monclient: dump: 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca12445640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:20.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.824+0000 7fca12445640 4 set_mon_vals no callback set 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 10 monclient: shutdown 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 1 -- 172.21.15.117:0/410317262 >> v1:172.21.15.117:6790/0 conn(0x7fca0c0b0500 legacy=0x7fca0c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 1 -- 172.21.15.117:0/410317262 shutdown_connections 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 1 -- 172.21.15.117:0/410317262 >> 172.21.15.117:0/410317262 conn(0x7fca0c06b660 msgr2=0x7fca0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 1 -- 172.21.15.117:0/410317262 shutdown_connections 2024-03-29T11:20:20.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.825+0000 7fca12445640 1 -- 172.21.15.117:0/410317262 wait complete. 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.826+0000 7fca12445640 10 monclient: build_initial_monmap 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.826+0000 7fca12445640 10 monclient: monmap: 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:20.813 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.827+0000 7fca12445640 1 librados: starting msgr at 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.827+0000 7fca12445640 1 librados: starting objecter 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.828+0000 7fca12445640 1 Processor -- start 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.828+0000 7fca12445640 1 -- start start 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.828+0000 7fca12445640 1 librados: setting wanted keys 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.828+0000 7fca12445640 1 librados: calling monclient init 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.828+0000 7fca12445640 10 monclient: init 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.829+0000 7fca12445640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:20.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.829+0000 7fca12445640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.829+0000 7fca12445640 10 monclient(hunting): picked mon.noname-c con 0x7fca0c0aeca0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.829+0000 7fca12445640 10 monclient(hunting): picked mon.noname-b con 0x7fca0c0b0500 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca12445640 10 monclient(hunting): picked mon.noname-a con 0x7fca0c070180 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca0b7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fca0c0b0500 0x7fca0c13aae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:51370/0 (socket says 172.21.15.117:51370) 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca0b7fe640 1 -- 172.21.15.117:0/333388183 learned_addr learned my addr 172.21.15.117:0/333388183 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c11ef40 con 0x7fca0c070180 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c1d7160 con 0x7fca0c0b0500 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca08ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c0b0500 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca08ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c0aeca0 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.830+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c11f170 con 0x7fca0c0aeca0 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca12445640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:20.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca12445640 10 monclient(hunting): authenticate will time out at 889.140202s 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 client.?.objecter ms_handle_connect 0x7fca0c070180 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501296354 0 0) 0x7fca0c1d7160 con 0x7fca0c0b0500 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8004eb0 con 0x7fca0c0b0500 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.831+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107495635 0 0) 0x7fca0c11f170 con 0x7fca0c0aeca0 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:20:20.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0c1d7160 con 0x7fca0c0aeca0 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512201286 0 0) 0x7fc9f8004eb0 con 0x7fca0c0b0500 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0c11f170 con 0x7fca0c0b0500 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252136405 0 0) 0x7fca0c11ef40 con 0x7fca0c070180 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8004eb0 con 0x7fca0c070180 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fca00001830 con 0x7fca0c0b0500 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.832+0000 7fca08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:20.817 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 455279248 0 0) 0x7fca0c11f170 con 0x7fca0c0b0500 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 >> v1:172.21.15.117:6790/0 conn(0x7fca0c0aeca0 legacy=0x7fca0c1d6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 >> v1:172.21.15.117:6789/0 conn(0x7fca0c070180 legacy=0x7fca0c1d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:20.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 monclient: found mon.b 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c1d7390 con 0x7fca0c0b0500 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 10 monclient: _renew_subs 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c13b1f0 con 0x7fca0c0b0500 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:20.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 10 monclient: _renew_subs 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c1d39a0 con 0x7fca0c0b0500 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca12445640 1 librados: init done 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca00001990 con 0x7fca0c0b0500 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fca00004980 con 0x7fca0c0b0500 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca08ff9640 10 monclient: dump: 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:20.820 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.833+0000 7fca0a7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.835+0000 7fca12445640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.835+0000 7fca12445640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.835+0000 7fca12445640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.835+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c0ad0c0 con 0x7fca0c0b0500 2024-03-29T11:20:20.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.836+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fca00004ae0 con 0x7fca0c0b0500 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.837+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fca000849f0 con 0x7fca0c0b0500 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.837+0000 7fca0bfff640 10 monclient: get_auth_request con 0x7fc9f806ddf0 auth_method 0 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.837+0000 7fca08ff9640 10 client.4279.objecter ms_dispatch 0x7fca0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.837+0000 7fca08ff9640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.837+0000 7fca08ff9640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:20.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.838+0000 7fca08ff9640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:20:20.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.838+0000 7fca08ff9640 10 client.4279.objecter ms_handle_connect 0x7fc9f806ddf0 2024-03-29T11:20:20.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.843+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca00050100 con 0x7fca0c0b0500 2024-03-29T11:20:20.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.843+0000 7fca08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:20.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:20.843+0000 7fca08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:21.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.234+0000 7fca12445640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.234+0000 7fca12445640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.234+0000 7fca12445640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:21.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.234+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca0c075f10 con 0x7fca0c0b0500 2024-03-29T11:20:21.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.235+0000 7fca08ff9640 1 -- 172.21.15.117:0/333388183 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fca00052200 con 0x7fca0c0b0500 2024-03-29T11:20:21.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.235+0000 7fca08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:21.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.235+0000 7fca08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:21.213 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:21.213 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 10 librados: watch_flush enter 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 10 librados: watch_flush exit 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 >> v1:172.21.15.117:6816/4276038175 conn(0x7fc9f806ddf0 legacy=0x7fc9f80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 10 monclient: shutdown 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:21.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 >> v1:172.21.15.173:6789/0 conn(0x7fca0c0b0500 legacy=0x7fca0c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:21.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.239+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 shutdown_connections 2024-03-29T11:20:21.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.240+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 >> 172.21.15.117:0/333388183 conn(0x7fca0c06b660 msgr2=0x7fca0c0b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:21.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.240+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 shutdown_connections 2024-03-29T11:20:21.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.240+0000 7fca12445640 1 -- 172.21.15.117:0/333388183 wait complete. 2024-03-29T11:20:21.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:21.240+0000 7fca12445640 1 librados: shutdown 2024-03-29T11:20:21.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:28.040 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:28.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.201+0000 7f1998425640 10 monclient: get_monmap_and_config 2024-03-29T11:20:28.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.201+0000 7f1998425640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.201+0000 7f1998425640 10 monclient: monmap: 2024-03-29T11:20:28.178 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:28.202790+0000 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:28.202790+0000 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:28.179 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.202+0000 7f1998425640 1 Processor -- start 2024-03-29T11:20:28.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.202+0000 7f1998425640 1 -- start start 2024-03-29T11:20:28.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.202+0000 7f1998425640 10 monclient: init 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient(hunting): picked mon.noname-b con 0x7f19901457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f19961c1640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f19901457f0 0x7f199013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44610/0 (socket says 172.21.15.117:44610) 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f19961c1640 1 -- 172.21.15.117:0/1649209799 learned_addr learned my addr 172.21.15.117:0/1649209799 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient(hunting): picked mon.noname-a con 0x7f199013f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient(hunting): picked mon.noname-c con 0x7f19900a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900aaef0 con 0x7f199013f650 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1990145bd0 con 0x7f19900a9d70 2024-03-29T11:20:28.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1990145eb0 con 0x7f19901457f0 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f1998425640 10 monclient(hunting): authenticate will time out at 896.513628s 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.204+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422903830 0 0) 0x7f1990145eb0 con 0x7f19901457f0 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c003520 con 0x7f19901457f0 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782850525 0 0) 0x7f197c003520 con 0x7f19901457f0 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 >> v1:172.21.15.173:6789/0 conn(0x7f19900a9d70 legacy=0x7f199014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.205+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 >> v1:172.21.15.117:6789/0 conn(0x7f199013f650 legacy=0x7f19900b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 1 monclient: found mon.noname-b 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990146340 con 0x7f19901457f0 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f198c001690 con 0x7f19901457f0 2024-03-29T11:20:28.186 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f1998425640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f1998425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f19951bf640 10 monclient: dump: 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:28.187 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.206+0000 7f1998425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f198c001920 con 0x7f19901457f0 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 1 -- 172.21.15.117:0/1649209799 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f198c001cb0 con 0x7f19901457f0 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f19951bf640 10 monclient: dump: 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.188 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f1998425640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.207+0000 7f1998425640 4 set_mon_vals no callback set 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.208+0000 7f1998425640 10 monclient: shutdown 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.208+0000 7f1998425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.208+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 >> v1:172.21.15.117:6790/0 conn(0x7f19901457f0 legacy=0x7f199013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.209+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 shutdown_connections 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.209+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 >> 172.21.15.117:0/1649209799 conn(0x7f1990066a00 msgr2=0x7f1990066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.209+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 shutdown_connections 2024-03-29T11:20:28.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.209+0000 7f1998425640 1 -- 172.21.15.117:0/1649209799 wait complete. 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.210+0000 7f1998425640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.210+0000 7f1998425640 10 monclient: monmap: 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:28.193 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 librados: starting msgr at 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 librados: starting objecter 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 Processor -- start 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 -- start start 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 librados: setting wanted keys 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 1 librados: calling monclient init 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.211+0000 7f1998425640 10 monclient: init 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient(hunting): picked mon.noname-a con 0x7f19900a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient(hunting): picked mon.noname-c con 0x7f199013f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f19959c0640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f199013f650 0x7f19901371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45936/0 (socket says 172.21.15.117:45936) 2024-03-29T11:20:28.194 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f19959c0640 1 -- 172.21.15.117:0/2687283153 learned_addr learned my addr 172.21.15.117:0/2687283153 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient(hunting): picked mon.noname-b con 0x7f19901457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900ac4b0 con 0x7f19900a9d70 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900ac790 con 0x7f19901457f0 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900ab600 con 0x7f199013f650 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 client.?.objecter ms_handle_connect 0x7f199013f650 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 client.?.objecter ms_handle_connect 0x7f19901457f0 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1998425640 10 monclient(hunting): authenticate will time out at 896.521753s 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819685592 0 0) 0x7f19900ac790 con 0x7f19901457f0 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1974004eb0 con 0x7f19901457f0 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187268567 0 0) 0x7f19900ab600 con 0x7f199013f650 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.212+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19900ac790 con 0x7f199013f650 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2770860067 0 0) 0x7f1974004eb0 con 0x7f19901457f0 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19900ab600 con 0x7f19901457f0 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1988001b70 con 0x7f19901457f0 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:28.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 514133301 0 0) 0x7f19900ac790 con 0x7f199013f650 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1974004eb0 con 0x7f199013f650 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1980004040 con 0x7f199013f650 2024-03-29T11:20:28.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.213+0000 7f1986ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.214+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148378980 0 0) 0x7f1974004eb0 con 0x7f199013f650 2024-03-29T11:20:28.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.214+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 >> v1:172.21.15.173:6789/0 conn(0x7f19901457f0 legacy=0x7f19901e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.214+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 >> v1:172.21.15.117:6789/0 conn(0x7f19900a9d70 legacy=0x7f1990142560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 1 monclient: found mon.c 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990141fc0 con 0x7f199013f650 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 10 monclient: _renew_subs 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1990142c70 con 0x7f199013f650 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:28.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 10 monclient: _renew_subs 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1980000ed0 con 0x7f199013f650 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f199013ff80 con 0x7f199013f650 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f19800012a0 con 0x7f199013f650 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1986ffd640 10 monclient: dump: 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:28.200 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f1998425640 1 librados: init done 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.215+0000 7f19949be640 4 set_mon_vals no callback set 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.216+0000 7f1998425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.216+0000 7f1998425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.216+0000 7f1998425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.216+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1978005080 con 0x7f199013f650 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.217+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f1980001400 con 0x7f199013f650 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f1980084a00 con 0x7f199013f650 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f1986ffd640 10 client.4292.objecter ms_dispatch 0x7f1990000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f1986ffd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f1986ffd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f1986ffd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.218+0000 7f19961c1640 10 monclient: get_auth_request con 0x7f197406dec0 auth_method 0 2024-03-29T11:20:28.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.219+0000 7f1986ffd640 10 client.4292.objecter ms_handle_connect 0x7f197406dec0 2024-03-29T11:20:28.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.225+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19800502e0 con 0x7f199013f650 2024-03-29T11:20:28.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.225+0000 7f1986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:28.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.225+0000 7f1986ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:28.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.532+0000 7f1998425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.532+0000 7f1998425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.532+0000 7f1998425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:28.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.532+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1978005330 con 0x7f199013f650 2024-03-29T11:20:28.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.533+0000 7f1986ffd640 1 -- 172.21.15.117:0/2687283153 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f1980052210 con 0x7f199013f650 2024-03-29T11:20:28.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.533+0000 7f1986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:28.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.533+0000 7f1986ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:28.511 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:28.511 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 10 librados: watch_flush enter 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 10 librados: watch_flush exit 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 >> v1:172.21.15.117:6816/4276038175 conn(0x7f197406dec0 legacy=0x7f1974070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 10 monclient: shutdown 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:28.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.536+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 >> v1:172.21.15.117:6790/0 conn(0x7f199013f650 legacy=0x7f19901371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:28.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.537+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 shutdown_connections 2024-03-29T11:20:28.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.537+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 >> 172.21.15.117:0/2687283153 conn(0x7f1990066a00 msgr2=0x7f1990066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:28.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.537+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 shutdown_connections 2024-03-29T11:20:28.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.537+0000 7f1998425640 1 -- 172.21.15.117:0/2687283153 wait complete. 2024-03-29T11:20:28.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:28.537+0000 7f1998425640 1 librados: shutdown 2024-03-29T11:20:28.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:35.335 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:35.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.507+0000 7f73a2fc6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.507+0000 7f73a2fc6640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.507+0000 7f73a2fc6640 10 monclient: monmap: 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:35.508956+0000 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:35.508956+0000 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:35.485 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.508+0000 7f73a2fc6640 1 Processor -- start 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 1 -- start start 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient: init 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f739c1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f739c13f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f739c0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c0aaef0 con 0x7f739c13f650 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c145bd0 con 0x7f739c0a9d70 2024-03-29T11:20:35.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c145eb0 con 0x7f739c1457f0 2024-03-29T11:20:35.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.509+0000 7f73a2fc6640 10 monclient(hunting): authenticate will time out at 903.818986s 2024-03-29T11:20:35.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.511+0000 7f73a1563640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f739c0a9d70 0x7f739c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53192/0 (socket says 172.21.15.117:53192) 2024-03-29T11:20:35.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.511+0000 7f73a1563640 1 -- 172.21.15.117:0/2892479495 learned_addr learned my addr 172.21.15.117:0/2892479495 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261984937 0 0) 0x7f739c145bd0 con 0x7f739c0a9d70 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:20:35.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.512+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c003520 con 0x7f739c0a9d70 2024-03-29T11:20:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504002472 0 0) 0x7f738c003520 con 0x7f739c0a9d70 2024-03-29T11:20:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 >> v1:172.21.15.117:6790/0 conn(0x7f739c1457f0 legacy=0x7f739c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:35.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 >> v1:172.21.15.117:6789/0 conn(0x7f739c13f650 legacy=0x7f739c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c146340 con 0x7f739c0a9d70 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7394001a10 con 0x7f739c0a9d70 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: dump: 2024-03-29T11:20:35.491 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f73a2fc6640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f73a2fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7394001be0 con 0x7f739c0a9d70 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 1 -- 172.21.15.117:0/2892479495 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7394001ff0 con 0x7f739c0a9d70 2024-03-29T11:20:35.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f739b7fe640 10 monclient: dump: 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f73a2fc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.513+0000 7f73a2fc6640 4 set_mon_vals no callback set 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.514+0000 7f73a2fc6640 10 monclient: shutdown 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.514+0000 7f73a2fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.514+0000 7f73a2fc6640 1 -- 172.21.15.117:0/2892479495 >> v1:172.21.15.173:6789/0 conn(0x7f739c0a9d70 legacy=0x7f739c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.515+0000 7f73a2fc6640 1 -- 172.21.15.117:0/2892479495 shutdown_connections 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.515+0000 7f73a2fc6640 1 -- 172.21.15.117:0/2892479495 >> 172.21.15.117:0/2892479495 conn(0x7f739c066a00 msgr2=0x7f739c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.515+0000 7f73a2fc6640 1 -- 172.21.15.117:0/2892479495 shutdown_connections 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.515+0000 7f73a2fc6640 1 -- 172.21.15.117:0/2892479495 wait complete. 2024-03-29T11:20:35.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 10 monclient: monmap: 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 librados: starting msgr at 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 librados: starting objecter 2024-03-29T11:20:35.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 Processor -- start 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 -- start start 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 librados: setting wanted keys 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 1 librados: calling monclient init 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.516+0000 7f73a2fc6640 10 monclient: init 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f739c0a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f739c13f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f739c1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c0abc60 con 0x7f739c13f650 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c06de90 con 0x7f739c1457f0 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739c1e7f10 con 0x7f739c0a9d70 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.518+0000 7f73a2fc6640 10 monclient(hunting): authenticate will time out at 903.827317s 2024-03-29T11:20:35.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.521+0000 7f73a1563640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f739c1457f0 0x7f739c1ef620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:54344/0 (socket says 172.21.15.117:54344) 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.521+0000 7f73a1563640 1 -- 172.21.15.117:0/967105218 learned_addr learned my addr 172.21.15.117:0/967105218 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.521+0000 7f73997fa640 10 client.?.objecter ms_handle_connect 0x7f739c1457f0 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.521+0000 7f73997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 client.?.objecter ms_handle_connect 0x7f739c0a9d70 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173219387 0 0) 0x7f739c06de90 con 0x7f739c1457f0 2024-03-29T11:20:35.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7380004eb0 con 0x7f739c1457f0 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 client.?.objecter ms_handle_connect 0x7f739c13f650 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944447453 0 0) 0x7f739c1e7f10 con 0x7f739c0a9d70 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739c06de90 con 0x7f739c0a9d70 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058250289 0 0) 0x7f739c0abc60 con 0x7f739c13f650 2024-03-29T11:20:35.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739c1e7f10 con 0x7f739c13f650 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852890944 0 0) 0x7f739c06de90 con 0x7f739c0a9d70 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7380006680 con 0x7f739c0a9d70 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7390001770 con 0x7f739c0a9d70 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): dump: 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.502 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3349035273 0 0) 0x7f739c1e7f10 con 0x7f739c13f650 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7380006d30 con 0x7f739c13f650 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f73880017e0 con 0x7f739c13f650 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient(hunting): dump: 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:35.503 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3716320122 0 0) 0x7f7380006680 con 0x7f739c0a9d70 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 >> v1:172.21.15.173:6789/0 conn(0x7f739c1457f0 legacy=0x7f739c1ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 >> v1:172.21.15.117:6789/0 conn(0x7f739c13f650 legacy=0x7f739c13bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 monclient: found mon.c 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c1e80a0 con 0x7f739c0a9d70 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.522+0000 7f73997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73a2fc6640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73a2fc6640 10 monclient: _renew_subs 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73a2fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:35.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73a2fc6640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739c070210 con 0x7f739c0a9d70 2024-03-29T11:20:35.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7390001c60 con 0x7f739c0a9d70 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7390001770 con 0x7f739c0a9d70 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f73997fa640 10 monclient: dump: 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:35.507 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.524+0000 7f73a2fc6640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.524+0000 7f73a2fc6640 10 monclient: _renew_subs 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.524+0000 7f73a2fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.524+0000 7f73a2fc6640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c1e76d0 con 0x7f739c0a9d70 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.524+0000 7f73a2fc6640 1 librados: init done 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.523+0000 7f739affd640 4 set_mon_vals no callback set 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.526+0000 7f7386ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.526+0000 7f7386ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.526+0000 7f7386ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.526+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739c075f10 con 0x7f739c0a9d70 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.526+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7390019460 con 0x7f739c0a9d70 2024-03-29T11:20:35.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.528+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f7390053220 con 0x7f739c0a9d70 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.528+0000 7f73997fa640 10 client.4295.objecter ms_dispatch 0x7f739c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.528+0000 7f73997fa640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.528+0000 7f73997fa640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.528+0000 7f73997fa640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.529+0000 7f739bfff640 10 monclient: get_auth_request con 0x7f738006e560 auth_method 0 2024-03-29T11:20:35.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.529+0000 7f73997fa640 10 client.4295.objecter ms_handle_connect 0x7f738006e560 2024-03-29T11:20:35.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.534+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f739004eb60 con 0x7f739c0a9d70 2024-03-29T11:20:35.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.534+0000 7f73997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.534+0000 7f73997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.868+0000 7f7386ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.868+0000 7f7386ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.868+0000 7f7386ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:35.845 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.868+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f739c0703a0 con 0x7f739c0a9d70 2024-03-29T11:20:35.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.870+0000 7f73997fa640 1 -- 172.21.15.117:0/967105218 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f7390050c60 con 0x7f739c0a9d70 2024-03-29T11:20:35.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.870+0000 7f73997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.870+0000 7f73997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.847 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:35.847 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:35.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.872+0000 7f7386ffd640 10 librados: watch_flush enter 2024-03-29T11:20:35.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.872+0000 7f7386ffd640 10 librados: watch_flush exit 2024-03-29T11:20:35.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.872+0000 7f7386ffd640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.872+0000 7f7386ffd640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 >> v1:172.21.15.117:6816/4276038175 conn(0x7f738006e560 legacy=0x7f7380070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 10 monclient: shutdown 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 >> v1:172.21.15.117:6790/0 conn(0x7f739c0a9d70 legacy=0x7f739c070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 shutdown_connections 2024-03-29T11:20:35.850 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 >> 172.21.15.117:0/967105218 conn(0x7f739c066a00 msgr2=0x7f739c144da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.851 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 shutdown_connections 2024-03-29T11:20:35.851 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.873+0000 7f7386ffd640 1 -- 172.21.15.117:0/967105218 wait complete. 2024-03-29T11:20:35.851 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:35.874+0000 7f7386ffd640 1 librados: shutdown 2024-03-29T11:20:35.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:42.675 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:42.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.855+0000 7f6c13e45640 10 monclient: get_monmap_and_config 2024-03-29T11:20:42.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.855+0000 7f6c13e45640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.855+0000 7f6c13e45640 10 monclient: monmap: 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:42.856301+0000 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:42.856301+0000 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:42.833 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:42.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.857+0000 7f6c13e45640 1 Processor -- start 2024-03-29T11:20:42.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.857+0000 7f6c13e45640 1 -- start start 2024-03-29T11:20:42.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.857+0000 7f6c13e45640 10 monclient: init 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c0a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-c con 0x7f6c0c13ee90 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0c0af900 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c0afd10 con 0x7f6c0c0a9ac0 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c0af6f0 con 0x7f6c0c13ee90 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c0aac80 con 0x7f6c0c0af900 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.835 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.858+0000 7f6c13e45640 10 monclient(hunting): authenticate will time out at 911.167682s 2024-03-29T11:20:42.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.861+0000 7f6c123e2640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6c0c0af900 0x7f6c0c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38907/0 (socket says 172.21.15.117:38907) 2024-03-29T11:20:42.838 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.861+0000 7f6c123e2640 1 -- 172.21.15.117:0/3051750763 learned_addr learned my addr 172.21.15.117:0/3051750763 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368192954 0 0) 0x7f6c0c0aac80 con 0x7f6c0c0af900 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf8003520 con 0x7f6c0c0af900 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443721381 0 0) 0x7f6c0c0af6f0 con 0x7f6c0c13ee90 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.840 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:20:42.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.862+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c0c0aac80 con 0x7f6c0c13ee90 2024-03-29T11:20:42.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025542665 0 0) 0x7f6bf8003520 con 0x7f6c0c0af900 2024-03-29T11:20:42.841 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 >> v1:172.21.15.173:6789/0 conn(0x7f6c0c13ee90 legacy=0x7f6c0c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 >> v1:172.21.15.117:6789/0 conn(0x7f6c0c0a9ac0 legacy=0x7f6c0c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:42.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 1 monclient: found mon.noname-b 2024-03-29T11:20:42.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.842 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0aaf60 con 0x7f6c0c0af900 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.865+0000 7f6c10bdf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c04001a10 con 0x7f6c0c0af900 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 10 monclient: dump: 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.843 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c13e45640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c13e45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c04001b70 con 0x7f6c0c0af900 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c10bdf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c13e45640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:42.844 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.866+0000 7f6c13e45640 4 set_mon_vals no callback set 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c13e45640 10 monclient: shutdown 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c13e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c13e45640 1 -- 172.21.15.117:0/3051750763 >> v1:172.21.15.117:6790/0 conn(0x7f6c0c0af900 legacy=0x7f6c0c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c10bdf640 1 -- 172.21.15.117:0/3051750763 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c04001fc0 con 0x7f6c0c0af900 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c10bdf640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c13e45640 1 -- 172.21.15.117:0/3051750763 shutdown_connections 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.867+0000 7f6c13e45640 1 -- 172.21.15.117:0/3051750763 >> 172.21.15.117:0/3051750763 conn(0x7f6c0c0b35e0 msgr2=0x7f6c0c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:42.846 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.869+0000 7f6c13e45640 1 -- 172.21.15.117:0/3051750763 shutdown_connections 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.870+0000 7f6c13e45640 1 -- 172.21.15.117:0/3051750763 wait complete. 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.871+0000 7f6c13e45640 10 monclient: build_initial_monmap 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.871+0000 7f6c13e45640 10 monclient: monmap: 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:42.848 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:42.849 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:42.849 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:42.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.871+0000 7f6c13e45640 1 librados: starting msgr at 2024-03-29T11:20:42.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.871+0000 7f6c13e45640 1 librados: starting objecter 2024-03-29T11:20:42.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.871+0000 7f6c13e45640 1 Processor -- start 2024-03-29T11:20:42.851 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.873+0000 7f6c13e45640 1 -- start start 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 1 librados: setting wanted keys 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 1 librados: calling monclient init 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient: init 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0c0a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-c con 0x7f6c0c0af900 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c13ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c072310 con 0x7f6c0c13ee90 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c071fa0 con 0x7f6c0c0a9ac0 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c1e3b10 con 0x7f6c0c0af900 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:42.852 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.874+0000 7f6c13e45640 10 monclient(hunting): authenticate will time out at 911.183854s 2024-03-29T11:20:42.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.878+0000 7f6c113e0640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6c0c0af900 0x7f6c0c1e2cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39011/0 (socket says 172.21.15.117:39011) 2024-03-29T11:20:42.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.878+0000 7f6c113e0640 1 -- 172.21.15.117:0/2151610153 learned_addr learned my addr 172.21.15.117:0/2151610153 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:42.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 client.?.objecter ms_handle_connect 0x7f6c0c0af900 2024-03-29T11:20:42.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122235447 0 0) 0x7f6c0c1e3b10 con 0x7f6c0c0af900 2024-03-29T11:20:42.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:42.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0004eb0 con 0x7f6c0c0af900 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300919796 0 0) 0x7f6bf0004eb0 con 0x7f6c0c0af900 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c0c1e3b10 con 0x7f6c0c0af900 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6bfc001840 con 0x7f6c0c0af900 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.879+0000 7f6c027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient(hunting): dump: 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.858 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 client.?.objecter ms_handle_connect 0x7f6c0c13ee90 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1793444024 0 0) 0x7f6c0c1e3b10 con 0x7f6c0c0af900 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 >> v1:172.21.15.173:6789/0 conn(0x7f6c0c0a9ac0 legacy=0x7f6c0c1e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 >> v1:172.21.15.117:6789/0 conn(0x7f6c0c13ee90 legacy=0x7f6c0c1e3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 monclient: found mon.c 2024-03-29T11:20:42.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0715c0 con 0x7f6c0c0af900 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 10 monclient: _renew_subs 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c1421d0 con 0x7f6c0c0af900 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:42.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 10 monclient: _renew_subs 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c070b00 con 0x7f6c0c0af900 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c13e45640 1 librados: init done 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bfc0019a0 con 0x7f6c0c0af900 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6bfc003810 con 0x7f6c0c0af900 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.880+0000 7f6c027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.881+0000 7f6c027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.881+0000 7f6c027fc640 10 monclient: dump: 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:42.861 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:42.862 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:42.862 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:42.862 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:42.862 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:42.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.882+0000 7f6c03fff640 4 set_mon_vals no callback set 2024-03-29T11:20:42.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.883+0000 7f6c13e45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.883+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f6bfc003a50 con 0x7f6c0c0af900 2024-03-29T11:20:42.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.883+0000 7f6c13e45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.883+0000 7f6c13e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.883+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf4005080 con 0x7f6c0c0af900 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f6bfc084a20 con 0x7f6c0c0af900 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c11be1640 10 monclient: get_auth_request con 0x7f6bf006dc60 auth_method 0 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c027fc640 10 client.4301.objecter ms_dispatch 0x7f6c0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c027fc640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c027fc640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.884+0000 7f6c027fc640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:20:42.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.886+0000 7f6c027fc640 10 client.4301.objecter ms_handle_connect 0x7f6bf006dc60 2024-03-29T11:20:42.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.893+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bfc050300 con 0x7f6c0c0af900 2024-03-29T11:20:42.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.893+0000 7f6c027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:42.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:42.893+0000 7f6c027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:43.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.234+0000 7f6c13e45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.234+0000 7f6c13e45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.234+0000 7f6c13e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.234+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bf4005330 con 0x7f6c0c0af900 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.234+0000 7f6c027fc640 1 -- 172.21.15.117:0/2151610153 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f6bfc052230 con 0x7f6c0c0af900 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.235+0000 7f6c027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:43.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.235+0000 7f6c027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:43.212 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:43.212 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:43.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.238+0000 7f6c13e45640 10 librados: watch_flush enter 2024-03-29T11:20:43.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.238+0000 7f6c13e45640 10 librados: watch_flush exit 2024-03-29T11:20:43.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.238+0000 7f6c13e45640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:20:43.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.238+0000 7f6c13e45640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 >> v1:172.21.15.117:6816/4276038175 conn(0x7f6bf006dc60 legacy=0x7f6bf0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 10 monclient: shutdown 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 >> v1:172.21.15.117:6790/0 conn(0x7f6c0c0af900 legacy=0x7f6c0c1e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 shutdown_connections 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 >> 172.21.15.117:0/2151610153 conn(0x7f6c0c0b35e0 msgr2=0x7f6c0c0acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 shutdown_connections 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 -- 172.21.15.117:0/2151610153 wait complete. 2024-03-29T11:20:43.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:43.239+0000 7f6c13e45640 1 librados: shutdown 2024-03-29T11:20:43.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:50.042 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.216+0000 7f2338e12640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.216+0000 7f2338e12640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.216+0000 7f2338e12640 10 monclient: monmap: 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:50.217730+0000 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:50.217730+0000 2024-03-29T11:20:50.193 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.194 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.194 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:50.194 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:50.194 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:50.194 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.220+0000 7f2338e12640 1 Processor -- start 2024-03-29T11:20:50.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.220+0000 7f2338e12640 1 -- start start 2024-03-29T11:20:50.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.220+0000 7f2338e12640 10 monclient: init 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-a con 0x7f232c038c60 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-c con 0x7f232c0324b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-b con 0x7f232c0d11e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0caf80 con 0x7f232c038c60 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c039850 con 0x7f232c0324b0 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0c78d0 con 0x7f232c0d11e0 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.221+0000 7f2338e12640 10 monclient(hunting): authenticate will time out at 918.530733s 2024-03-29T11:20:50.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.226+0000 7f2332ffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f232c0324b0 0x7f232c0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:34395/0 (socket says 172.21.15.117:34395) 2024-03-29T11:20:50.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.226+0000 7f2332ffd640 1 -- 172.21.15.117:0/1715316580 learned_addr learned my addr 172.21.15.117:0/1715316580 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:50.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953072212 0 0) 0x7f232c039850 con 0x7f232c0324b0 2024-03-29T11:20:50.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:20:50.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.228+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324003520 con 0x7f232c0324b0 2024-03-29T11:20:50.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676343694 0 0) 0x7f232c0c78d0 con 0x7f232c0d11e0 2024-03-29T11:20:50.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c039850 con 0x7f232c0d11e0 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183997833 0 0) 0x7f232c0caf80 con 0x7f232c038c60 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23240039d0 con 0x7f232c038c60 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 458736133 0 0) 0x7f232c039850 con 0x7f232c0d11e0 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 >> v1:172.21.15.173:6789/0 conn(0x7f232c0324b0 legacy=0x7f232c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 >> v1:172.21.15.117:6789/0 conn(0x7f232c038c60 legacy=0x7f232c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 monclient: found mon.noname-b 2024-03-29T11:20:50.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0c7a60 con 0x7f232c0d11e0 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f233404f730 con 0x7f232c0d11e0 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.230+0000 7f23327fc640 10 monclient: dump: 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.211 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f2338e12640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f2338e12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23340008d0 con 0x7f232c0d11e0 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f23327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f2338e12640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.231+0000 7f2338e12640 4 set_mon_vals no callback set 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f23327fc640 1 -- 172.21.15.117:0/1715316580 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f233404f730 con 0x7f232c0d11e0 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f23327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f23327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f23327fc640 10 monclient: dump: 2024-03-29T11:20:50.212 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 10 monclient: shutdown 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 1 -- 172.21.15.117:0/1715316580 >> v1:172.21.15.117:6790/0 conn(0x7f232c0d11e0 legacy=0x7f232c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 1 -- 172.21.15.117:0/1715316580 shutdown_connections 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 1 -- 172.21.15.117:0/1715316580 >> 172.21.15.117:0/1715316580 conn(0x7f232c005c20 msgr2=0x7f232c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 1 -- 172.21.15.117:0/1715316580 shutdown_connections 2024-03-29T11:20:50.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.232+0000 7f2338e12640 1 -- 172.21.15.117:0/1715316580 wait complete. 2024-03-29T11:20:50.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.233+0000 7f2338e12640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.233+0000 7f2338e12640 10 monclient: monmap: 2024-03-29T11:20:50.215 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:50.215 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.233+0000 7f2338e12640 1 librados: starting msgr at 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.233+0000 7f2338e12640 1 librados: starting objecter 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.233+0000 7f2338e12640 1 Processor -- start 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.234+0000 7f2338e12640 1 -- start start 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.234+0000 7f2338e12640 1 librados: setting wanted keys 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.234+0000 7f2338e12640 1 librados: calling monclient init 2024-03-29T11:20:50.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.234+0000 7f2338e12640 10 monclient: init 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.235+0000 7f2338e12640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.235+0000 7f2338e12640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.235+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-a con 0x7f232c0324b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.238+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-c con 0x7f232c038c60 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.239+0000 7f2332ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f232c038c60 0x7f232c0bce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:55405/0 (socket says 172.21.15.117:55405) 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.239+0000 7f2332ffd640 1 -- 172.21.15.117:0/3309979410 learned_addr learned my addr 172.21.15.117:0/3309979410 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.239+0000 7f231ffff640 10 client.?.objecter ms_handle_connect 0x7f232c038c60 2024-03-29T11:20:50.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.239+0000 7f231ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 10 monclient(hunting): picked mon.noname-b con 0x7f232c0d11e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:50.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0bdc40 con 0x7f232c0324b0 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0343f0 con 0x7f232c0d11e0 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0359e0 con 0x7f232c038c60 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f2338e12640 10 monclient(hunting): authenticate will time out at 918.551326s 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 client.?.objecter ms_handle_connect 0x7f232c0d11e0 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036517288 0 0) 0x7f232c0343f0 con 0x7f232c0d11e0 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:20:50.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.242+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318004eb0 con 0x7f232c0d11e0 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188998276 0 0) 0x7f2318004eb0 con 0x7f232c0d11e0 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f232c0343f0 con 0x7f232c0d11e0 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f233404f730 con 0x7f232c0d11e0 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.221 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.222 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:50.222 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:50.222 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:50.222 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.244+0000 7f231ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1325001615 0 0) 0x7f232c0343f0 con 0x7f232c0d11e0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 >> v1:172.21.15.117:6790/0 conn(0x7f232c038c60 legacy=0x7f232c0bce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 >> v1:172.21.15.117:6789/0 conn(0x7f232c0324b0 legacy=0x7f232c15ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 1 monclient: found mon.b 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c034bb0 con 0x7f232c0d11e0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 client.?.objecter ms_handle_connect 0x7f232c0324b0 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f231ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 10 monclient: _renew_subs 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c0c5290 con 0x7f232c0d11e0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 10 monclient: _renew_subs 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c0c4af0 con 0x7f232c0d11e0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.245+0000 7f2338e12640 1 librados: init done 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f233404fdd0 con 0x7f232c0d11e0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f23340688d0 con 0x7f232c0d11e0 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.246+0000 7f231ffff640 10 monclient: dump: 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.247+0000 7f2338e12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.247+0000 7f2331ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f2334067d90 con 0x7f232c0d11e0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f2338e12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f2338e12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232c007b70 con 0x7f232c0d11e0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f23340fe230 con 0x7f232c0d11e0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 10 client.4300.objecter ms_dispatch 0x7f2334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.248+0000 7f231ffff640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.249+0000 7f23337fe640 10 monclient: get_auth_request con 0x7f231806daf0 auth_method 0 2024-03-29T11:20:50.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.249+0000 7f231ffff640 10 client.4300.objecter ms_handle_connect 0x7f231806daf0 2024-03-29T11:20:50.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.257+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23340c9b10 con 0x7f232c0d11e0 2024-03-29T11:20:50.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.257+0000 7f231ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.257+0000 7f231ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.586+0000 7f2338e12640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.586+0000 7f2338e12640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.586+0000 7f2338e12640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:20:50.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.586+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f232c034820 con 0x7f232c0d11e0 2024-03-29T11:20:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.588+0000 7f231ffff640 1 -- 172.21.15.117:0/3309979410 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f23340cba40 con 0x7f232c0d11e0 2024-03-29T11:20:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.588+0000 7f231ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:50.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.588+0000 7f231ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.567 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:50.567 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:50.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.592+0000 7f2338e12640 10 librados: watch_flush enter 2024-03-29T11:20:50.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.593+0000 7f2338e12640 10 librados: watch_flush exit 2024-03-29T11:20:50.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.593+0000 7f2338e12640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.593+0000 7f2338e12640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 >> v1:172.21.15.117:6816/4276038175 conn(0x7f231806daf0 legacy=0x7f231806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 10 monclient: shutdown 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 >> v1:172.21.15.173:6789/0 conn(0x7f232c0d11e0 legacy=0x7f232c0bd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 shutdown_connections 2024-03-29T11:20:50.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.595+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 >> 172.21.15.117:0/3309979410 conn(0x7f232c005c20 msgr2=0x7f232c03bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.596+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 shutdown_connections 2024-03-29T11:20:50.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.596+0000 7f2338e12640 1 -- 172.21.15.117:0/3309979410 wait complete. 2024-03-29T11:20:50.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:50.596+0000 7f2338e12640 1 librados: shutdown 2024-03-29T11:20:50.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:57.402 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.577+0000 7f4c3440e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.577+0000 7f4c3440e640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.577+0000 7f4c3440e640 10 monclient: monmap: 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:20:57.578930+0000 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:20:57.578930+0000 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.556 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.578+0000 7f4c3440e640 1 Processor -- start 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.579+0000 7f4c3440e640 1 -- start start 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.579+0000 7f4c3440e640 10 monclient: init 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c0b4100 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c13ee90 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0af6f0 con 0x7f4c2c0b4100 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0aac80 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0aaf60 con 0x7f4c2c13ee90 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.580+0000 7f4c3440e640 10 monclient(hunting): authenticate will time out at 925.889212s 2024-03-29T11:20:57.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.586+0000 7f4c321aa640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4c2c0b4100 0x7f4c2c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47949/0 (socket says 172.21.15.117:47949) 2024-03-29T11:20:57.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.586+0000 7f4c321aa640 1 -- 172.21.15.117:0/684584716 learned_addr learned my addr 172.21.15.117:0/684584716 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:57.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.588+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279605078 0 0) 0x7f4c2c0aaf60 con 0x7f4c2c13ee90 2024-03-29T11:20:57.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c18003520 con 0x7f4c2c13ee90 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779143867 0 0) 0x7f4c2c0af6f0 con 0x7f4c2c0b4100 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c0aaf60 con 0x7f4c2c0b4100 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238277689 0 0) 0x7f4c2c0aac80 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2c0af6f0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3832115192 0 0) 0x7f4c2c0aaf60 con 0x7f4c2c0b4100 2024-03-29T11:20:57.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 >> v1:172.21.15.117:6790/0 conn(0x7f4c2c13ee90 legacy=0x7f4c2c146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 >> v1:172.21.15.173:6789/0 conn(0x7f4c2c0a9ac0 legacy=0x7f4c2c14a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 monclient: found mon.noname-a 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c13e970 con 0x7f4c2c0b4100 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4c280015f0 con 0x7f4c2c0b4100 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.589+0000 7f4c311a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.590+0000 7f4c311a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.590+0000 7f4c311a8640 10 monclient: dump: 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:57.570 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.590+0000 7f4c3440e640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:20:57.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.590+0000 7f4c3440e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c28001750 con 0x7f4c2c0b4100 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 1 -- 172.21.15.117:0/684584716 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4c28001b60 con 0x7f4c2c0b4100 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:57.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.592+0000 7f4c311a8640 10 monclient: dump: 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.593+0000 7f4c3440e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.593+0000 7f4c3440e640 4 set_mon_vals no callback set 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.594+0000 7f4c3440e640 10 monclient: shutdown 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.594+0000 7f4c3440e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.594+0000 7f4c3440e640 1 -- 172.21.15.117:0/684584716 >> v1:172.21.15.117:6789/0 conn(0x7f4c2c0b4100 legacy=0x7f4c2c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.596+0000 7f4c3440e640 1 -- 172.21.15.117:0/684584716 shutdown_connections 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.596+0000 7f4c3440e640 1 -- 172.21.15.117:0/684584716 >> 172.21.15.117:0/684584716 conn(0x7f4c2c0b35e0 msgr2=0x7f4c2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.597+0000 7f4c3440e640 1 -- 172.21.15.117:0/684584716 shutdown_connections 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.598+0000 7f4c3440e640 1 -- 172.21.15.117:0/684584716 wait complete. 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 10 monclient: build_initial_monmap 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 10 monclient: monmap: 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:20:57.576 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 librados: starting msgr at 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 librados: starting objecter 2024-03-29T11:20:57.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 Processor -- start 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 -- start start 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 librados: setting wanted keys 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 1 librados: calling monclient init 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.599+0000 7f4c3440e640 10 monclient: init 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c0a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2c0b4100 addr v1:172.21.15.173:6789/0 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c13ee90 addr v1:172.21.15.117:6790/0 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c135a10 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c133df0 con 0x7f4c2c0b4100 2024-03-29T11:20:57.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c13b9e0 con 0x7f4c2c13ee90 2024-03-29T11:20:57.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:57.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.600+0000 7f4c3440e640 10 monclient(hunting): authenticate will time out at 925.909699s 2024-03-29T11:20:57.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c321aa640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4c2c0a9ac0 0x7f4c2c072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:48165/0 (socket says 172.21.15.117:48165) 2024-03-29T11:20:57.581 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c321aa640 1 -- 172.21.15.117:0/404894693 learned_addr learned my addr 172.21.15.117:0/404894693 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:20:57.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 client.?.objecter ms_handle_connect 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164887108 0 0) 0x7f4c2c135a10 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.605+0000 7f4c22ffd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10004eb0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 250503130 0 0) 0x7f4c10004eb0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c2c135a10 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4c28001b60 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:57.584 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.606+0000 7f4c22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.607+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1768776352 0 0) 0x7f4c2c135a10 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.607+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 >> v1:172.21.15.117:6790/0 conn(0x7f4c2c13ee90 legacy=0x7f4c2c13a210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 >> v1:172.21.15.173:6789/0 conn(0x7f4c2c0b4100 legacy=0x7f4c2c1e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 1 monclient: found mon.a 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c13af40 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:57.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 client.?.objecter ms_handle_connect 0x7f4c2c0b4100 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c3440e640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c3440e640 10 monclient: _renew_subs 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c3440e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.608+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c13b1f0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c3440e640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c3440e640 10 monclient: _renew_subs 2024-03-29T11:20:57.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c3440e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c13ab00 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c3440e640 1 librados: init done 2024-03-29T11:20:57.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c28004ad0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:57.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4c28004ee0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:57.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:57.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.609+0000 7f4c22ffd640 10 monclient: dump: 2024-03-29T11:20:57.592 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.610+0000 7f4c3440e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.610+0000 7f4c3440e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.610+0000 7f4c3440e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.610+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c134030 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.611+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4c2801aeb0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c309a7640 4 set_mon_vals no callback set 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f4c28084ce0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c22ffd640 10 client.4302.objecter ms_dispatch 0x7f4c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c22ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c22ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c22ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.612+0000 7f4c319a9640 10 monclient: get_auth_request con 0x7f4c1006dbc0 auth_method 0 2024-03-29T11:20:57.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.613+0000 7f4c22ffd640 10 client.4302.objecter ms_handle_connect 0x7f4c1006dbc0 2024-03-29T11:20:57.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.621+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c2c134030 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.621+0000 7f4c22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:57.598 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.621+0000 7f4c22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:57.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.946+0000 7f4c3440e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.946+0000 7f4c3440e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.946+0000 7f4c3440e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:20:57.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.946+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c2c075f10 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.947+0000 7f4c22ffd640 1 -- 172.21.15.117:0/404894693 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f4c280503f0 con 0x7f4c2c0a9ac0 2024-03-29T11:20:57.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.947+0000 7f4c22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:57.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.947+0000 7f4c22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:57.924 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:20:57.924 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 10 librados: watch_flush enter 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 10 librados: watch_flush exit 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 >> v1:172.21.15.117:6816/4276038175 conn(0x7f4c1006dbc0 legacy=0x7f4c10070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 10 monclient: shutdown 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:57.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.951+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 >> v1:172.21.15.117:6789/0 conn(0x7f4c2c0a9ac0 legacy=0x7f4c2c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:57.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.953+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 shutdown_connections 2024-03-29T11:20:57.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.953+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 >> 172.21.15.117:0/404894693 conn(0x7f4c2c0b35e0 msgr2=0x7f4c2c0ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:57.930 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.954+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 shutdown_connections 2024-03-29T11:20:57.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.954+0000 7f4c3440e640 1 -- 172.21.15.117:0/404894693 wait complete. 2024-03-29T11:20:57.931 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:20:57.954+0000 7f4c3440e640 1 librados: shutdown 2024-03-29T11:20:57.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:04.754 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:04.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.927+0000 7f538f5c8640 10 monclient: get_monmap_and_config 2024-03-29T11:21:04.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.927+0000 7f538f5c8640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.927+0000 7f538f5c8640 10 monclient: monmap: 2024-03-29T11:21:04.905 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:04.928426+0000 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:04.928426+0000 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.928+0000 7f538f5c8640 1 Processor -- start 2024-03-29T11:21:04.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.928+0000 7f538f5c8640 1 -- start start 2024-03-29T11:21:04.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.928+0000 7f538f5c8640 10 monclient: init 2024-03-29T11:21:04.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:04.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-b con 0x7f53880a96c0 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:04.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-a con 0x7f53880a9aa0 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-c con 0x7f53880ad730 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538813fee0 con 0x7f53880a9aa0 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880aab80 con 0x7f53880ad730 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5388145e80 con 0x7f53880a96c0 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.929+0000 7f538f5c8640 10 monclient(hunting): authenticate will time out at 933.238932s 2024-03-29T11:21:04.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.932+0000 7f538d364640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f53880a96c0 0x7f538813f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:35127/0 (socket says 172.21.15.117:35127) 2024-03-29T11:21:04.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.932+0000 7f538d364640 1 -- 172.21.15.117:0/272825979 learned_addr learned my addr 172.21.15.117:0/272825979 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:04.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207031074 0 0) 0x7f5388145e80 con 0x7f53880a96c0 2024-03-29T11:21:04.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:21:04.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.933+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5374003520 con 0x7f53880a96c0 2024-03-29T11:21:04.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718854902 0 0) 0x7f53880aab80 con 0x7f53880ad730 2024-03-29T11:21:04.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388145e80 con 0x7f53880ad730 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840860594 0 0) 0x7f5374003520 con 0x7f53880a96c0 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 >> v1:172.21.15.173:6789/0 conn(0x7f53880ad730 legacy=0x7f538814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.936+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 >> v1:172.21.15.117:6789/0 conn(0x7f53880a9aa0 legacy=0x7f5388145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 1 monclient: found mon.noname-b 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880b4440 con 0x7f53880a96c0 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5378001610 con 0x7f53880a96c0 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f538f5c8640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: dump: 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.918 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f538f5c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5378001770 con 0x7f53880a96c0 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f538f5c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 1 -- 172.21.15.117:0/272825979 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5378001bc0 con 0x7f53880a96c0 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f538f5c8640 4 set_mon_vals no callback set 2024-03-29T11:21:04.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.937+0000 7f537ffff640 10 monclient: dump: 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.938+0000 7f538f5c8640 10 monclient: shutdown 2024-03-29T11:21:04.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.938+0000 7f538f5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:04.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.938+0000 7f538f5c8640 1 -- 172.21.15.117:0/272825979 >> v1:172.21.15.117:6790/0 conn(0x7f53880a96c0 legacy=0x7f538813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.941+0000 7f538f5c8640 1 -- 172.21.15.117:0/272825979 shutdown_connections 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.941+0000 7f538f5c8640 1 -- 172.21.15.117:0/272825979 >> 172.21.15.117:0/272825979 conn(0x7f53880b35e0 msgr2=0x7f53880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.941+0000 7f538f5c8640 1 -- 172.21.15.117:0/272825979 shutdown_connections 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.942+0000 7f538f5c8640 1 -- 172.21.15.117:0/272825979 wait complete. 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.943+0000 7f538f5c8640 10 monclient: build_initial_monmap 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.943+0000 7f538f5c8640 10 monclient: monmap: 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:04.922 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.943+0000 7f538f5c8640 1 librados: starting msgr at 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.943+0000 7f538f5c8640 1 librados: starting objecter 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.943+0000 7f538f5c8640 1 Processor -- start 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.944+0000 7f538f5c8640 1 -- start start 2024-03-29T11:21:04.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.944+0000 7f538f5c8640 1 librados: setting wanted keys 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.944+0000 7f538f5c8640 1 librados: calling monclient init 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.944+0000 7f538f5c8640 10 monclient: init 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-a con 0x7f53880a96c0 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-b con 0x7f53880ad730 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient(hunting): picked mon.noname-c con 0x7f538806fd30 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880717c0 con 0x7f53880a96c0 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5388071130 con 0x7f53880ad730 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53881d7440 con 0x7f538806fd30 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:04.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.945+0000 7f538f5c8640 10 monclient(hunting): authenticate will time out at 933.254766s 2024-03-29T11:21:04.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f538cb63640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f53880ad730 0x7f5388071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:44731/0 (socket says 172.21.15.117:44731) 2024-03-29T11:21:04.928 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f538cb63640 1 -- 172.21.15.117:0/2663252039 learned_addr learned my addr 172.21.15.117:0/2663252039 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:04.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter ms_handle_connect 0x7f538806fd30 2024-03-29T11:21:04.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.935 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter ms_handle_connect 0x7f53880ad730 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter ms_handle_connect 0x7f53880a96c0 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121154318 0 0) 0x7f5388071130 con 0x7f53880ad730 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c004eb0 con 0x7f53880ad730 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152621448 0 0) 0x7f53880717c0 con 0x7f53880a96c0 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.936 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.952+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388071130 con 0x7f53880a96c0 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897291195 0 0) 0x7f53881d7440 con 0x7f538806fd30 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53880717c0 con 0x7f538806fd30 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.953+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4267620281 0 0) 0x7f53880717c0 con 0x7f538806fd30 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53881d7440 con 0x7f538806fd30 2024-03-29T11:21:04.937 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5380001cb0 con 0x7f538806fd30 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient(hunting): dump: 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:04.938 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2064642188 0 0) 0x7f53881d7440 con 0x7f538806fd30 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 >> v1:172.21.15.173:6789/0 conn(0x7f53880ad730 legacy=0x7f5388071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 >> v1:172.21.15.117:6789/0 conn(0x7f53880a96c0 legacy=0x7f538806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 monclient: found mon.c 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880713a0 con 0x7f538806fd30 2024-03-29T11:21:04.939 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.954+0000 7f537dffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.955+0000 7f537dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.955+0000 7f537dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.955+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5380003990 con 0x7f538806fd30 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.955+0000 7f537dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.955+0000 7f537effd640 4 set_mon_vals no callback set 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5380003ca0 con 0x7f538806fd30 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f537dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f537dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f537dffb640 10 monclient: dump: 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:04.940 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 10 monclient: _renew_subs 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5388072340 con 0x7f538806fd30 2024-03-29T11:21:04.941 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 10 monclient: _renew_subs 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53881d7fa0 con 0x7f538806fd30 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.956+0000 7f538f5c8640 1 librados: init done 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.958+0000 7f538f5c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.958+0000 7f538f5c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.958+0000 7f538f5c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.958+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388071a80 con 0x7f538806fd30 2024-03-29T11:21:04.942 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.958+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f5380019c70 con 0x7f538806fd30 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f5380083ac0 con 0x7f538806fd30 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f537dffb640 10 client.4316.objecter ms_dispatch 0x7f5388000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f537dffb640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f537dffb640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f537dffb640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.959+0000 7f538d364640 10 monclient: get_auth_request con 0x7f536c06df20 auth_method 0 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.960+0000 7f537dffb640 10 client.4316.objecter ms_handle_connect 0x7f536c06df20 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.966+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5380085e60 con 0x7f538806fd30 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.966+0000 7f537dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:04.944 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:04.966+0000 7f537dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:05.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.298+0000 7f538f5c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.298+0000 7f538f5c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.298+0000 7f538f5c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:05.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.298+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5388075f10 con 0x7f538806fd30 2024-03-29T11:21:05.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.300+0000 7f537dffb640 1 -- 172.21.15.117:0/2663252039 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f538004f1d0 con 0x7f538806fd30 2024-03-29T11:21:05.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.300+0000 7f537dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:05.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.300+0000 7f537dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:05.277 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:05.277 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:05.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.303+0000 7f538f5c8640 10 librados: watch_flush enter 2024-03-29T11:21:05.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.303+0000 7f538f5c8640 10 librados: watch_flush exit 2024-03-29T11:21:05.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.303+0000 7f538f5c8640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:21:05.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.303+0000 7f538f5c8640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:21:05.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 >> v1:172.21.15.117:6816/4276038175 conn(0x7f536c06df20 legacy=0x7f536c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 10 monclient: shutdown 2024-03-29T11:21:05.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 >> v1:172.21.15.117:6790/0 conn(0x7f538806fd30 legacy=0x7f53881d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 shutdown_connections 2024-03-29T11:21:05.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.304+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 >> 172.21.15.117:0/2663252039 conn(0x7f53880b35e0 msgr2=0x7f53880b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.305+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 shutdown_connections 2024-03-29T11:21:05.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.307+0000 7f538f5c8640 1 -- 172.21.15.117:0/2663252039 wait complete. 2024-03-29T11:21:05.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:05.307+0000 7f538f5c8640 1 librados: shutdown 2024-03-29T11:21:05.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:12.108 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:12.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.285+0000 7f9b06cf1640 10 monclient: get_monmap_and_config 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.285+0000 7f9b06cf1640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.285+0000 7f9b06cf1640 10 monclient: monmap: 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:12.286375+0000 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:12.286375+0000 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:12.262 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.286+0000 7f9b06cf1640 1 Processor -- start 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.286+0000 7f9b06cf1640 1 -- start start 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.286+0000 7f9b06cf1640 10 monclient: init 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f9b0006bfa0 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f9b0006c380 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:12.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f9b0014a050 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:12.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0006c870 con 0x7f9b0006bfa0 2024-03-29T11:21:12.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0006dc10 con 0x7f9b0006c380 2024-03-29T11:21:12.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0006cfd0 con 0x7f9b0014a050 2024-03-29T11:21:12.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.287+0000 7f9b06cf1640 10 monclient(hunting): authenticate will time out at 940.596385s 2024-03-29T11:21:12.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.290+0000 7f9b0528e640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9b0014a050 0x7f9b00151ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44777/0 (socket says 172.21.15.117:44777) 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.293+0000 7f9afffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9b0006c380 0x7f9b00071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:44313/0 (socket says 172.21.15.117:44313) 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.293+0000 7f9afffff640 1 -- 172.21.15.117:0/1608718049 learned_addr learned my addr 172.21.15.117:0/1608718049 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475715985 0 0) 0x7f9b0006dc10 con 0x7f9b0006c380 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae8003520 con 0x7f9b0006c380 2024-03-29T11:21:12.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1451894988 0 0) 0x7f9ae8003520 con 0x7f9b0006c380 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 >> v1:172.21.15.117:6790/0 conn(0x7f9b0014a050 legacy=0x7f9b00151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 >> v1:172.21.15.117:6789/0 conn(0x7f9b0006bfa0 legacy=0x7f9b00149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b00071f50 con 0x7f9b0006c380 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.272 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9af0001a10 con 0x7f9b0006c380 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.294+0000 7f9aff7fe640 10 monclient: dump: 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.273 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9af0001ca0 con 0x7f9b0006c380 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 1 -- 172.21.15.117:0/1608718049 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9af0002030 con 0x7f9b0006c380 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.295+0000 7f9aff7fe640 10 monclient: dump: 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:12.274 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 4 set_mon_vals no callback set 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 10 monclient: shutdown 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.275 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.296+0000 7f9b06cf1640 1 -- 172.21.15.117:0/1608718049 >> v1:172.21.15.173:6789/0 conn(0x7f9b0006c380 legacy=0x7f9b00071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.298+0000 7f9b06cf1640 1 -- 172.21.15.117:0/1608718049 shutdown_connections 2024-03-29T11:21:12.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.298+0000 7f9b06cf1640 1 -- 172.21.15.117:0/1608718049 >> 172.21.15.117:0/1608718049 conn(0x7f9b00067340 msgr2=0x7f9b000b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.299+0000 7f9b06cf1640 1 -- 172.21.15.117:0/1608718049 shutdown_connections 2024-03-29T11:21:12.276 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.300+0000 7f9b06cf1640 1 -- 172.21.15.117:0/1608718049 wait complete. 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.301+0000 7f9b06cf1640 10 monclient: build_initial_monmap 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.301+0000 7f9b06cf1640 10 monclient: monmap: 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:12.278 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.301+0000 7f9b06cf1640 1 librados: starting msgr at 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.301+0000 7f9b06cf1640 1 librados: starting objecter 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.301+0000 7f9b06cf1640 1 Processor -- start 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.302+0000 7f9b06cf1640 1 -- start start 2024-03-29T11:21:12.279 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.302+0000 7f9b06cf1640 1 librados: setting wanted keys 2024-03-29T11:21:12.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.302+0000 7f9b06cf1640 1 librados: calling monclient init 2024-03-29T11:21:12.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.302+0000 7f9b06cf1640 10 monclient: init 2024-03-29T11:21:12.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:12.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:12.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f9b0006bfa0 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:12.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f9b0006c380 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:12.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f9b0014a050 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:12.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000722c0 con 0x7f9b0014a050 2024-03-29T11:21:12.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000aa280 con 0x7f9b0006c380 2024-03-29T11:21:12.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000a94e0 con 0x7f9b0006bfa0 2024-03-29T11:21:12.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:12.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.303+0000 7f9b06cf1640 10 monclient(hunting): authenticate will time out at 940.612369s 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.306+0000 7f9afffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9b0006c380 0x7f9b000aa7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:44445/0 (socket says 172.21.15.117:44445) 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.306+0000 7f9afffff640 1 -- 172.21.15.117:0/958578709 learned_addr learned my addr 172.21.15.117:0/958578709 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.306+0000 7f9afd7fa640 10 client.?.objecter ms_handle_connect 0x7f9b0006c380 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.306+0000 7f9afd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847283846 0 0) 0x7f9b000aa280 con 0x7f9b0006c380 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:21:12.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.307+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec004eb0 con 0x7f9b0006c380 2024-03-29T11:21:12.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72429948 0 0) 0x7f9aec004eb0 con 0x7f9b0006c380 2024-03-29T11:21:12.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b000aa280 con 0x7f9b0006c380 2024-03-29T11:21:12.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9af00018d0 con 0x7f9b0006c380 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 10 monclient(hunting): dump: 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.287 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.309+0000 7f9afd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 client.?.objecter ms_handle_connect 0x7f9b0014a050 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 client.?.objecter ms_handle_connect 0x7f9b0006bfa0 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906877220 0 0) 0x7f9b000a94e0 con 0x7f9b0006bfa0 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.291 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec004eb0 con 0x7f9b0006bfa0 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340881510 0 0) 0x7f9b000722c0 con 0x7f9b0014a050 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b000a94e0 con 0x7f9b0014a050 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733615868 0 0) 0x7f9b000aa280 con 0x7f9b0006c380 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 >> v1:172.21.15.117:6790/0 conn(0x7f9b0006bfa0 legacy=0x7f9b000aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 >> v1:172.21.15.117:6789/0 conn(0x7f9b0014a050 legacy=0x7f9b001e7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 monclient: found mon.b 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:12.292 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000a9670 con 0x7f9b0006c380 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9afd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 10 monclient: _renew_subs 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b001df910 con 0x7f9b0006c380 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 10 monclient: _renew_subs 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b000a9f40 con 0x7f9b0006c380 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.311+0000 7f9b06cf1640 1 librados: init done 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9af0004f70 con 0x7f9b0006c380 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:12.293 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9af0005280 con 0x7f9b0006c380 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.312+0000 7f9afd7fa640 10 monclient: dump: 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:12.294 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.313+0000 7f9b06cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.313+0000 7f9b06cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.313+0000 7f9b06cf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.313+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b000aa410 con 0x7f9b0006c380 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.313+0000 7f9afeffd640 4 set_mon_vals no callback set 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.315+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f9af001b250 con 0x7f9b0006c380 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.316+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f9af0084f10 con 0x7f9b0006c380 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.316+0000 7f9afd7fa640 10 client.4318.objecter ms_dispatch 0x7f9b00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:12.295 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.316+0000 7f9afd7fa640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:12.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.316+0000 7f9afd7fa640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:12.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.316+0000 7f9afd7fa640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T11:21:12.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.317+0000 7f9b04a8d640 10 monclient: get_auth_request con 0x7f9aec06de50 auth_method 0 2024-03-29T11:21:12.296 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.320+0000 7f9afd7fa640 10 client.4318.objecter ms_handle_connect 0x7f9aec06de50 2024-03-29T11:21:12.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.326+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9af00507f0 con 0x7f9b0006c380 2024-03-29T11:21:12.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.326+0000 7f9afd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:12.303 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.326+0000 7f9afd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:12.619 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.643+0000 7f9b06cf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.643+0000 7f9b06cf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.643+0000 7f9b06cf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.643+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b0006d160 con 0x7f9b0006c380 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.644+0000 7f9afd7fa640 1 -- 172.21.15.117:0/958578709 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f9af0052720 con 0x7f9b0006c380 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.644+0000 7f9afd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:12.620 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.644+0000 7f9afd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:12.621 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:12.621 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:12.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.649+0000 7f9b06cf1640 10 librados: watch_flush enter 2024-03-29T11:21:12.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.649+0000 7f9b06cf1640 10 librados: watch_flush exit 2024-03-29T11:21:12.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.649+0000 7f9b06cf1640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T11:21:12.626 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.649+0000 7f9b06cf1640 10 client.4318.objecter successfully canceled tick 2024-03-29T11:21:12.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.651+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 >> v1:172.21.15.117:6816/4276038175 conn(0x7f9aec06de50 legacy=0x7f9aec070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.651+0000 7f9b06cf1640 10 monclient: shutdown 2024-03-29T11:21:12.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.651+0000 7f9b06cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:12.627 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.651+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 >> v1:172.21.15.173:6789/0 conn(0x7f9b0006c380 legacy=0x7f9b000aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:12.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.653+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 shutdown_connections 2024-03-29T11:21:12.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.653+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 >> 172.21.15.117:0/958578709 conn(0x7f9b00067340 msgr2=0x7f9b000b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:12.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.653+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 shutdown_connections 2024-03-29T11:21:12.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.655+0000 7f9b06cf1640 1 -- 172.21.15.117:0/958578709 wait complete. 2024-03-29T11:21:12.631 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:12.655+0000 7f9b06cf1640 1 librados: shutdown 2024-03-29T11:21:12.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:19.457 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:19.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.648+0000 7fdcccf0f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:19.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.648+0000 7fdcccf0f640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.625 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.648+0000 7fdcccf0f640 10 monclient: monmap: 2024-03-29T11:21:19.625 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:19.626 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:19.626 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:19.649225+0000 2024-03-29T11:21:19.626 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:19.649225+0000 2024-03-29T11:21:19.626 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.626 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.627 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:19.627 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:19.627 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:19.627 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.650+0000 7fdcccf0f640 1 Processor -- start 2024-03-29T11:21:19.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.650+0000 7fdcccf0f640 1 -- start start 2024-03-29T11:21:19.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.650+0000 7fdcccf0f640 10 monclient: init 2024-03-29T11:21:19.628 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc80b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc8071eb0 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc8143d10 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc806ef80 con 0x7fdcc80b1f60 2024-03-29T11:21:19.629 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc806f260 con 0x7fdcc8071eb0 2024-03-29T11:21:19.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80b40c0 con 0x7fdcc8143d10 2024-03-29T11:21:19.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.630 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.651+0000 7fdcccf0f640 10 monclient(hunting): authenticate will time out at 947.960601s 2024-03-29T11:21:19.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.655+0000 7fdcc659c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdcc80b1f60 0x7fdcc8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58253/0 (socket says 172.21.15.117:58253) 2024-03-29T11:21:19.632 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.655+0000 7fdcc659c640 1 -- 172.21.15.117:0/4009344907 learned_addr learned my addr 172.21.15.117:0/4009344907 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:19.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576034245 0 0) 0x7fdcc80b40c0 con 0x7fdcc8143d10 2024-03-29T11:21:19.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb0003520 con 0x7fdcc8143d10 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374890534 0 0) 0x7fdcc806ef80 con 0x7fdcc80b1f60 2024-03-29T11:21:19.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:19.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.656+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc80b40c0 con 0x7fdcc80b1f60 2024-03-29T11:21:19.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951913869 0 0) 0x7fdcc806f260 con 0x7fdcc8071eb0 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc806ef80 con 0x7fdcc8071eb0 2024-03-29T11:21:19.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2772055624 0 0) 0x7fdcb0003520 con 0x7fdcc8143d10 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 >> v1:172.21.15.173:6789/0 conn(0x7fdcc8071eb0 legacy=0x7fdcc8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 >> v1:172.21.15.117:6789/0 conn(0x7fdcc80b1f60 legacy=0x7fdcc8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 monclient: found mon.noname-b 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:21:19.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8155d70 con 0x7fdcc8143d10 2024-03-29T11:21:19.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcc0001a10 con 0x7fdcc8143d10 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcc559a640 10 monclient: dump: 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:19.658 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:19.659 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:19.660 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcccf0f640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:21:19.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.657+0000 7fdcccf0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:19.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc0001b70 con 0x7fdcc8143d10 2024-03-29T11:21:19.660 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 1 -- 172.21.15.117:0/4009344907 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcc0001f80 con 0x7fdcc8143d10 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.658+0000 7fdcc559a640 10 monclient: dump: 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:19.661 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:19.662 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.662 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.662 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.662 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.662 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:19.663 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:19.663 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:19.663 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.661+0000 7fdcccf0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:19.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.661+0000 7fdcccf0f640 4 set_mon_vals no callback set 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.662+0000 7fdcccf0f640 10 monclient: shutdown 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.662+0000 7fdcccf0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.662+0000 7fdcccf0f640 1 -- 172.21.15.117:0/4009344907 >> v1:172.21.15.117:6790/0 conn(0x7fdcc8143d10 legacy=0x7fdcc8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.663+0000 7fdcccf0f640 1 -- 172.21.15.117:0/4009344907 shutdown_connections 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.663+0000 7fdcccf0f640 1 -- 172.21.15.117:0/4009344907 >> 172.21.15.117:0/4009344907 conn(0x7fdcc8066a00 msgr2=0x7fdcc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:19.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.663+0000 7fdcccf0f640 1 -- 172.21.15.117:0/4009344907 shutdown_connections 2024-03-29T11:21:19.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.663+0000 7fdcccf0f640 1 -- 172.21.15.117:0/4009344907 wait complete. 2024-03-29T11:21:19.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.664+0000 7fdcccf0f640 10 monclient: build_initial_monmap 2024-03-29T11:21:19.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.664+0000 7fdcccf0f640 10 monclient: monmap: 2024-03-29T11:21:19.665 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:19.665 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:19.666 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:19.667 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:19.667 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 librados: starting msgr at 2024-03-29T11:21:19.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 librados: starting objecter 2024-03-29T11:21:19.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 Processor -- start 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 -- start start 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 librados: setting wanted keys 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 1 librados: calling monclient init 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.665+0000 7fdcccf0f640 10 monclient: init 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:19.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:19.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc80b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:19.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc8143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:19.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc8139b40 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:19.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80ad210 con 0x7fdcc80b1f60 2024-03-29T11:21:19.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc813d2c0 con 0x7fdcc8143d10 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc813d7e0 con 0x7fdcc8139b40 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.666+0000 7fdcccf0f640 10 monclient(hunting): authenticate will time out at 947.975366s 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.672+0000 7fdcc5d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fdcc8143d10 0x7fdcc8139430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:34627/0 (socket says 172.21.15.117:34627) 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.672+0000 7fdcc5d9b640 1 -- 172.21.15.117:0/2480484007 learned_addr learned my addr 172.21.15.117:0/2480484007 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:19.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 client.?.objecter ms_handle_connect 0x7fdcc8143d10 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974573523 0 0) 0x7fdcc813d2c0 con 0x7fdcc8143d10 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.671 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:21:19.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8004eb0 con 0x7fdcc8143d10 2024-03-29T11:21:19.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 client.?.objecter ms_handle_connect 0x7fdcc8139b40 2024-03-29T11:21:19.672 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.673+0000 7fdcb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856983242 0 0) 0x7fdcc813d7e0 con 0x7fdcc8139b40 2024-03-29T11:21:19.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:19.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:19.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:19.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.677+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc813d2c0 con 0x7fdcc8139b40 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 client.?.objecter ms_handle_connect 0x7fdcc80b1f60 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268764301 0 0) 0x7fdca8004eb0 con 0x7fdcc8143d10 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc813d7e0 con 0x7fdcc8143d10 2024-03-29T11:21:19.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcbc001aa0 con 0x7fdcc8143d10 2024-03-29T11:21:19.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:19.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:19.680 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:19.680 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:19.681 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:19.682 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:19.682 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.679+0000 7fdcb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:19.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1175217298 0 0) 0x7fdcc813d7e0 con 0x7fdcc8143d10 2024-03-29T11:21:19.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 >> v1:172.21.15.117:6790/0 conn(0x7fdcc8139b40 legacy=0x7fdcc8132b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 >> v1:172.21.15.117:6789/0 conn(0x7fdcc80b1f60 legacy=0x7fdcc813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 1 monclient: found mon.b 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:19.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc8070910 con 0x7fdcc8143d10 2024-03-29T11:21:19.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:19.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:19.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:19.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcccf0f640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:21:19.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcccf0f640 10 monclient: _renew_subs 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcccf0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.680+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc8070b60 con 0x7fdcc8143d10 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.681+0000 7fdcccf0f640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.681+0000 7fdcccf0f640 10 monclient: _renew_subs 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.681+0000 7fdcccf0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:19.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.681+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc8133440 con 0x7fdcc8143d10 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.681+0000 7fdcccf0f640 1 librados: init done 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.682+0000 7fdcccf0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.682+0000 7fdcccf0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.682+0000 7fdcccf0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.682+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc813d450 con 0x7fdcc8143d10 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcbc001d00 con 0x7fdcc8143d10 2024-03-29T11:21:19.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcbc003a60 con 0x7fdcc8143d10 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcb6ffd640 10 monclient: dump: 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:19.687 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.683+0000 7fdcc4d99640 4 set_mon_vals no callback set 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.684+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fdcbc003c40 con 0x7fdcc8143d10 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.688+0000 7fdcc659c640 10 monclient: get_auth_request con 0x7fdca806e010 auth_method 0 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 10 client.4324.objecter ms_handle_connect 0x7fdca806e010 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fdcbc054740 con 0x7fdcc8143d10 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 10 client.4324.objecter ms_dispatch 0x7fdcc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcbc0509e0 con 0x7fdcc8143d10 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:19.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:19.689+0000 7fdcb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:19.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.022+0000 7fdcccf0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.022+0000 7fdcccf0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:19.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.022+0000 7fdcccf0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:19.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.022+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc8154cb0 con 0x7fdcc8143d10 2024-03-29T11:21:20.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.024+0000 7fdcb6ffd640 1 -- 172.21.15.117:0/2480484007 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fdcbc0509e0 con 0x7fdcc8143d10 2024-03-29T11:21:20.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.024+0000 7fdcb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:20.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.024+0000 7fdcb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:20.001 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:20.001 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.027+0000 7fdcccf0f640 10 librados: watch_flush enter 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 10 librados: watch_flush exit 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 >> v1:172.21.15.117:6816/4276038175 conn(0x7fdca806e010 legacy=0x7fdca80704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 10 monclient: shutdown 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 >> v1:172.21.15.173:6789/0 conn(0x7fdcc8143d10 legacy=0x7fdcc8139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 shutdown_connections 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 >> 172.21.15.117:0/2480484007 conn(0x7fdcc8066a00 msgr2=0x7fdcc81469f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 shutdown_connections 2024-03-29T11:21:20.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 -- 172.21.15.117:0/2480484007 wait complete. 2024-03-29T11:21:20.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:20.028+0000 7fdcccf0f640 1 librados: shutdown 2024-03-29T11:21:20.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:26.827 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:26.975 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.998+0000 7f36e94ba640 10 monclient: get_monmap_and_config 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.998+0000 7f36e94ba640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.998+0000 7f36e94ba640 10 monclient: monmap: 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:26.999692+0000 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:26.999692+0000 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.976 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.999+0000 7f36e94ba640 1 Processor -- start 2024-03-29T11:21:26.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.999+0000 7f36e94ba640 1 -- start start 2024-03-29T11:21:26.977 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:26.999+0000 7f36e94ba640 10 monclient: init 2024-03-29T11:21:26.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:26.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-a con 0x7f36e40ae250 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:26.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-b con 0x7f36e413f970 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:26.978 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-c con 0x7f36e4142800 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40afe00 con 0x7f36e40ae250 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e4142c10 con 0x7f36e4142800 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40afb20 con 0x7f36e413f970 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e94ba640 10 monclient(hunting): authenticate will time out at 955.309510s 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e2ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f36e40ae250 0x7f36e413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42048/0 (socket says 172.21.15.117:42048) 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.000+0000 7f36e2ffd640 1 -- 172.21.15.117:0/1715666346 learned_addr learned my addr 172.21.15.117:0/1715666346 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753383986 0 0) 0x7f36e40afe00 con 0x7f36e40ae250 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36cc003520 con 0x7f36e40ae250 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656248479 0 0) 0x7f36e40afb20 con 0x7f36e413f970 2024-03-29T11:21:26.979 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e40afe00 con 0x7f36e413f970 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928814799 0 0) 0x7f36cc003520 con 0x7f36e40ae250 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 >> v1:172.21.15.117:6790/0 conn(0x7f36e413f970 legacy=0x7f36e41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 >> v1:172.21.15.173:6789/0 conn(0x7f36e4142800 legacy=0x7f36e414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e40aab80 con 0x7f36e40ae250 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.980 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e94ba640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e94ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36d4001650 con 0x7f36e40ae250 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.001+0000 7f36e1ffb640 10 monclient: dump: 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:26.981 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e94ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36d40018a0 con 0x7f36e40ae250 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e94ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 1 -- 172.21.15.117:0/1715666346 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36d4001c30 con 0x7f36e40ae250 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e94ba640 4 set_mon_vals no callback set 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.002+0000 7f36e1ffb640 10 monclient: dump: 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.985 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 10 monclient: shutdown 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 1 -- 172.21.15.117:0/1715666346 >> v1:172.21.15.117:6789/0 conn(0x7f36e40ae250 legacy=0x7f36e413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 1 -- 172.21.15.117:0/1715666346 shutdown_connections 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 1 -- 172.21.15.117:0/1715666346 >> 172.21.15.117:0/1715666346 conn(0x7f36e40b35e0 msgr2=0x7f36e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 1 -- 172.21.15.117:0/1715666346 shutdown_connections 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.003+0000 7f36e94ba640 1 -- 172.21.15.117:0/1715666346 wait complete. 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.004+0000 7f36e94ba640 10 monclient: build_initial_monmap 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.004+0000 7f36e94ba640 10 monclient: monmap: 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:26.986 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.005+0000 7f36e94ba640 1 librados: starting msgr at 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.005+0000 7f36e94ba640 1 librados: starting objecter 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.005+0000 7f36e94ba640 1 Processor -- start 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 -- start start 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 librados: setting wanted keys 2024-03-29T11:21:26.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 librados: calling monclient init 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient: init 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-b con 0x7f36e40ae250 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-c con 0x7f36e413f970 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient(hunting): picked mon.noname-a con 0x7f36e4142800 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e413c730 con 0x7f36e4142800 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e41368a0 con 0x7f36e40ae250 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e4144300 con 0x7f36e413f970 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.006+0000 7f36e94ba640 10 monclient(hunting): authenticate will time out at 955.315919s 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36e37fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f36e4142800 0x7f36e4136190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42336/0 (socket says 172.21.15.117:42336) 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36e37fe640 1 -- 172.21.15.117:0/747447698 learned_addr learned my addr 172.21.15.117:0/747447698 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 client.?.objecter ms_handle_connect 0x7f36e4142800 2024-03-29T11:21:26.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 client.?.objecter ms_handle_connect 0x7f36e413f970 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309934760 0 0) 0x7f36e413c730 con 0x7f36e4142800 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d0004eb0 con 0x7f36e4142800 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404032365 0 0) 0x7f36e4144300 con 0x7f36e413f970 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.007+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e413c730 con 0x7f36e413f970 2024-03-29T11:21:26.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 client.?.objecter ms_handle_connect 0x7f36e40ae250 2024-03-29T11:21:26.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:26.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794305952 0 0) 0x7f36e41368a0 con 0x7f36e40ae250 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4144300 con 0x7f36e40ae250 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826751390 0 0) 0x7f36d0004eb0 con 0x7f36e4142800 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.008+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e41368a0 con 0x7f36e4142800 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1040382661 0 0) 0x7f36e413c730 con 0x7f36e413f970 2024-03-29T11:21:26.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d0004eb0 con 0x7f36e413f970 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36dc001c30 con 0x7f36e4142800 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.994 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36d80018b0 con 0x7f36e413f970 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.995 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2536153541 0 0) 0x7f36e4144300 con 0x7f36e40ae250 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e413c730 con 0x7f36e40ae250 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36d4001bb0 con 0x7f36e40ae250 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:26.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2829301405 0 0) 0x7f36d0004eb0 con 0x7f36e413f970 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 >> v1:172.21.15.173:6789/0 conn(0x7f36e40ae250 legacy=0x7f36e413c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 >> v1:172.21.15.117:6789/0 conn(0x7f36e4142800 legacy=0x7f36e4136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:26.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 monclient: found mon.c 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e41445e0 con 0x7f36e413f970 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 10 monclient: _renew_subs 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e4143470 con 0x7f36e413f970 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 10 monclient: _renew_subs 2024-03-29T11:21:26.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e40acef0 con 0x7f36e413f970 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.009+0000 7f36e94ba640 1 librados: init done 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.010+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36d80035f0 con 0x7f36e413f970 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.010+0000 7f36cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f36d8003900 con 0x7f36e413f970 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36e17fa640 4 set_mon_vals no callback set 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36cb7fe640 10 monclient: dump: 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:26.999 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.011+0000 7f36e94ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.012+0000 7f36e94ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.012+0000 7f36e94ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.012+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4005080 con 0x7f36e413f970 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.014+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f36d80198d0 con 0x7f36e413f970 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36e2ffd640 10 monclient: get_auth_request con 0x7f36d006e110 auth_method 0 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f36d8083770 con 0x7f36e413f970 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 10 client.4334.objecter ms_dispatch 0x7f36e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:27.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:21:27.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.015+0000 7f36cb7fe640 10 client.4334.objecter ms_handle_connect 0x7f36d006e110 2024-03-29T11:21:27.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.021+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36d804f050 con 0x7f36e413f970 2024-03-29T11:21:27.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.021+0000 7f36cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:27.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.021+0000 7f36cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:27.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.395+0000 7f36e94ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.395+0000 7f36e94ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.395+0000 7f36e94ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:27.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.395+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36c4005330 con 0x7f36e413f970 2024-03-29T11:21:27.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.396+0000 7f36cb7fe640 1 -- 172.21.15.117:0/747447698 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f36d8056ba0 con 0x7f36e413f970 2024-03-29T11:21:27.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.396+0000 7f36cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:27.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.396+0000 7f36cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:27.373 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:27.373 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:27.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 10 librados: watch_flush enter 2024-03-29T11:21:27.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 10 librados: watch_flush exit 2024-03-29T11:21:27.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:21:27.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:21:27.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 >> v1:172.21.15.117:6816/4276038175 conn(0x7f36d006e110 legacy=0x7f36d00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 10 monclient: shutdown 2024-03-29T11:21:27.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:27.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 >> v1:172.21.15.117:6790/0 conn(0x7f36e413f970 legacy=0x7f36e4135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:27.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 shutdown_connections 2024-03-29T11:21:27.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 >> 172.21.15.117:0/747447698 conn(0x7f36e40b35e0 msgr2=0x7f36e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:27.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 shutdown_connections 2024-03-29T11:21:27.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.400+0000 7f36e94ba640 1 -- 172.21.15.117:0/747447698 wait complete. 2024-03-29T11:21:27.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:27.401+0000 7f36e94ba640 1 librados: shutdown 2024-03-29T11:21:27.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:34.200 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:34.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 10 monclient: get_monmap_and_config 2024-03-29T11:21:34.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 10 monclient: monmap: 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:34.362114+0000 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:34.362114+0000 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:34.338 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 1 Processor -- start 2024-03-29T11:21:34.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 1 -- start start 2024-03-29T11:21:34.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.361+0000 7facc72a1640 10 monclient: init 2024-03-29T11:21:34.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.362+0000 7facc72a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.362+0000 7facc72a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:34.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.362+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-c con 0x7facc01457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:34.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.362+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-b con 0x7facc013f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:34.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.362+0000 7facc503d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7facc01457f0 0x7facc013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:60370/0 (socket says 172.21.15.117:60370) 2024-03-29T11:21:34.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.363+0000 7facc503d640 1 -- 172.21.15.117:0/2674297610 learned_addr learned my addr 172.21.15.117:0/2674297610 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-a con 0x7facc00a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00aaef0 con 0x7facc00a9d70 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0145bd0 con 0x7facc01457f0 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0145eb0 con 0x7facc013f650 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facc72a1640 10 monclient(hunting): authenticate will time out at 962.673387s 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474656487 0 0) 0x7facc0145eb0 con 0x7facc013f650 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:21:34.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac003520 con 0x7facc013f650 2024-03-29T11:21:34.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3364055248 0 0) 0x7facac003520 con 0x7facc013f650 2024-03-29T11:21:34.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 >> v1:172.21.15.173:6789/0 conn(0x7facc01457f0 legacy=0x7facc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:34.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.364+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 >> v1:172.21.15.117:6789/0 conn(0x7facc00a9d70 legacy=0x7facc014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 1 monclient: found mon.noname-b 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0146340 con 0x7facc013f650 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7facb0001690 con 0x7facc013f650 2024-03-29T11:21:34.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.366+0000 7facb7fff640 10 monclient: dump: 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facc72a1640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:21:34.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facc72a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:34.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facb00018f0 con 0x7facc013f650 2024-03-29T11:21:34.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facc72a1640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facc72a1640 4 set_mon_vals no callback set 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 1 -- 172.21.15.117:0/2674297610 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7facb0001d40 con 0x7facc013f650 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.367+0000 7facb7fff640 10 monclient: dump: 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:34.346 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.368+0000 7facc72a1640 10 monclient: shutdown 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.368+0000 7facc72a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.368+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 >> v1:172.21.15.117:6790/0 conn(0x7facc013f650 legacy=0x7facc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.369+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 shutdown_connections 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.369+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 >> 172.21.15.117:0/2674297610 conn(0x7facc0066a00 msgr2=0x7facc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.369+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 shutdown_connections 2024-03-29T11:21:34.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.369+0000 7facc72a1640 1 -- 172.21.15.117:0/2674297610 wait complete. 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 10 monclient: build_initial_monmap 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 10 monclient: monmap: 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:34.348 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 1 librados: starting msgr at 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 1 librados: starting objecter 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 1 Processor -- start 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.370+0000 7facc72a1640 1 -- start start 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.371+0000 7facc72a1640 1 librados: setting wanted keys 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.371+0000 7facc72a1640 1 librados: calling monclient init 2024-03-29T11:21:34.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.371+0000 7facc72a1640 10 monclient: init 2024-03-29T11:21:34.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-c con 0x7facc00a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-a con 0x7facc013f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient(hunting): picked mon.noname-b con 0x7facc01457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00ac260 con 0x7facc013f650 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0137a20 con 0x7facc01457f0 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00ac4d0 con 0x7facc00a9d70 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.372+0000 7facc72a1640 10 monclient(hunting): authenticate will time out at 962.681780s 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facc503d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7facc00a9d70 0x7facc01425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:52016/0 (socket says 172.21.15.117:52016) 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facc503d640 1 -- 172.21.15.117:0/4209540523 learned_addr learned my addr 172.21.15.117:0/4209540523 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:34.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 client.?.objecter ms_handle_connect 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230715054 0 0) 0x7facc00ac4d0 con 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca4004eb0 con 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 872533891 0 0) 0x7faca4004eb0 con 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc00ac4d0 con 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 client.?.objecter ms_handle_connect 0x7facc01457f0 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7facbc0017d0 con 0x7facc00a9d70 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.373+0000 7facb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:34.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3499633251 0 0) 0x7facc00ac4d0 con 0x7facc00a9d70 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 >> v1:172.21.15.173:6789/0 conn(0x7facc01457f0 legacy=0x7facc0137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 >> v1:172.21.15.117:6789/0 conn(0x7facc013f650 legacy=0x7facc0136c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 1 monclient: found mon.c 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00ab620 con 0x7facc00a9d70 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 10 monclient: _renew_subs 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc0142ce0 con 0x7facc00a9d70 2024-03-29T11:21:34.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 10 monclient: _renew_subs 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facc013ff80 con 0x7facc00a9d70 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.374+0000 7facc72a1640 1 librados: init done 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facbc001930 con 0x7facc00a9d70 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7facbc004a80 con 0x7facc00a9d70 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb5ffb640 10 monclient: dump: 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:34.356 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:34.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.375+0000 7facb77fe640 4 set_mon_vals no callback set 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.376+0000 7facc72a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.376+0000 7facc72a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.376+0000 7facc72a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.376+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca0005080 con 0x7facc00a9d70 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.376+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7facbc004be0 con 0x7facc00a9d70 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.377+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7facbc084a40 con 0x7facc00a9d70 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.377+0000 7facb5ffb640 10 client.4340.objecter ms_dispatch 0x7facc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.377+0000 7facb5ffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.377+0000 7facb5ffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:34.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.377+0000 7facb5ffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:21:34.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.379+0000 7facc483c640 10 monclient: get_auth_request con 0x7faca406dbc0 auth_method 0 2024-03-29T11:21:34.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.379+0000 7facb5ffb640 10 client.4340.objecter ms_handle_connect 0x7faca406dbc0 2024-03-29T11:21:34.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.382+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facbc050380 con 0x7facc00a9d70 2024-03-29T11:21:34.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.382+0000 7facb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:34.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.382+0000 7facb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:34.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.685+0000 7facc72a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facc72a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facc72a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:34.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faca0005330 con 0x7facc00a9d70 2024-03-29T11:21:34.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facb5ffb640 1 -- 172.21.15.117:0/4209540523 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7facbc0522b0 con 0x7facc00a9d70 2024-03-29T11:21:34.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:34.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.686+0000 7facb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:34.663 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:34.663 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:34.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 10 librados: watch_flush enter 2024-03-29T11:21:34.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 10 librados: watch_flush exit 2024-03-29T11:21:34.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:21:34.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 >> v1:172.21.15.117:6816/4276038175 conn(0x7faca406dbc0 legacy=0x7faca4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 10 monclient: shutdown 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 >> v1:172.21.15.117:6790/0 conn(0x7facc00a9d70 legacy=0x7facc01425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 shutdown_connections 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 >> 172.21.15.117:0/4209540523 conn(0x7facc0066a00 msgr2=0x7facc00b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 shutdown_connections 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 -- 172.21.15.117:0/4209540523 wait complete. 2024-03-29T11:21:34.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:34.689+0000 7facc72a1640 1 librados: shutdown 2024-03-29T11:21:34.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:41.487 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:41.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.659+0000 7f9fa8484640 10 monclient: get_monmap_and_config 2024-03-29T11:21:41.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.659+0000 7f9fa8484640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.636 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.659+0000 7f9fa8484640 10 monclient: monmap: 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:41.660820+0000 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:41.660820+0000 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:41.637 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.661+0000 7f9fa8484640 1 Processor -- start 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.661+0000 7f9fa8484640 1 -- start start 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.661+0000 7f9fa8484640 10 monclient: init 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.638 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:41.639 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa00b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa0071eb0 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa0143d10 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa006ef80 con 0x7f9fa00b1f60 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa006f260 con 0x7f9fa0071eb0 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00b40c0 con 0x7f9fa0143d10 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.662+0000 7f9fa8484640 10 monclient(hunting): authenticate will time out at 969.971446s 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.663+0000 7f9fa6a21640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9fa0143d10 0x7f9fa0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40380/0 (socket says 172.21.15.117:40380) 2024-03-29T11:21:41.640 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.663+0000 7f9fa6a21640 1 -- 172.21.15.117:0/2367903032 learned_addr learned my addr 172.21.15.117:0/2367903032 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580994709 0 0) 0x7f9fa00b40c0 con 0x7f9fa0143d10 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c003520 con 0x7f9fa0143d10 2024-03-29T11:21:41.642 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567703825 0 0) 0x7f9fa006ef80 con 0x7f9fa00b1f60 2024-03-29T11:21:41.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:41.643 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa00b40c0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4153598495 0 0) 0x7f9f8c003520 con 0x7f9fa0143d10 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 >> v1:172.21.15.173:6789/0 conn(0x7f9fa0071eb0 legacy=0x7f9fa0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 >> v1:172.21.15.117:6789/0 conn(0x7f9fa00b1f60 legacy=0x7f9fa0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 monclient: found mon.noname-b 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.646 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0155d70 con 0x7f9fa0143d10 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f9c001690 con 0x7f9fa0143d10 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa521e640 10 monclient: dump: 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.647 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa8484640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.666+0000 7f9fa8484640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f9c0017f0 con 0x7f9fa0143d10 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa8484640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa8484640 4 set_mon_vals no callback set 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 1 -- 172.21.15.117:0/2367903032 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f9c001c00 con 0x7f9fa0143d10 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa521e640 10 monclient: dump: 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa8484640 10 monclient: shutdown 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa8484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.668+0000 7f9fa8484640 1 -- 172.21.15.117:0/2367903032 >> v1:172.21.15.117:6790/0 conn(0x7f9fa0143d10 legacy=0x7f9fa0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.669+0000 7f9fa8484640 1 -- 172.21.15.117:0/2367903032 shutdown_connections 2024-03-29T11:21:41.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.669+0000 7f9fa8484640 1 -- 172.21.15.117:0/2367903032 >> 172.21.15.117:0/2367903032 conn(0x7f9fa0066a00 msgr2=0x7f9fa0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:41.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.672+0000 7f9fa8484640 1 -- 172.21.15.117:0/2367903032 shutdown_connections 2024-03-29T11:21:41.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.672+0000 7f9fa8484640 1 -- 172.21.15.117:0/2367903032 wait complete. 2024-03-29T11:21:41.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.673+0000 7f9fa8484640 10 monclient: build_initial_monmap 2024-03-29T11:21:41.650 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.673+0000 7f9fa8484640 10 monclient: monmap: 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.673+0000 7f9fa8484640 1 librados: starting msgr at 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.673+0000 7f9fa8484640 1 librados: starting objecter 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.673+0000 7f9fa8484640 1 Processor -- start 2024-03-29T11:21:41.651 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 -- start start 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 librados: setting wanted keys 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 librados: calling monclient init 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient: init 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa00b1f60 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:41.652 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa0143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa00aa010 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00abbf0 con 0x7f9fa00aa010 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00abe40 con 0x7f9fa0143d10 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa01407b0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:41.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.674+0000 7f9fa8484640 10 monclient(hunting): authenticate will time out at 969.983902s 2024-03-29T11:21:41.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.678+0000 7f9fa6220640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9fa00b1f60 0x7f9fa0145240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41000/0 (socket says 172.21.15.117:41000) 2024-03-29T11:21:41.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.678+0000 7f9fa6220640 1 -- 172.21.15.117:0/4052997736 learned_addr learned my addr 172.21.15.117:0/4052997736 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:41.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 client.?.objecter ms_handle_connect 0x7f9fa00b1f60 2024-03-29T11:21:41.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784173968 0 0) 0x7f9fa01407b0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f84004eb0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 client.?.objecter ms_handle_connect 0x7f9fa0143d10 2024-03-29T11:21:41.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734850584 0 0) 0x7f9fa00abe40 con 0x7f9fa0143d10 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa01407b0 con 0x7f9fa0143d10 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2726364312 0 0) 0x7f9f84004eb0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f84005490 con 0x7f9fa00b1f60 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f98001cc0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:41.658 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:41.659 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:41.661 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.679+0000 7f9f96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132563919 0 0) 0x7f9f84005490 con 0x7f9fa00b1f60 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 >> v1:172.21.15.173:6789/0 conn(0x7f9fa0143d10 legacy=0x7f9fa013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 >> v1:172.21.15.117:6789/0 conn(0x7f9fa00aa010 legacy=0x7f9fa01400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 monclient: found mon.c 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa00ac120 con 0x7f9fa00b1f60 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:41.662 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f98003990 con 0x7f9fa00b1f60 2024-03-29T11:21:41.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:41.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.680+0000 7f9f97fff640 4 set_mon_vals no callback set 2024-03-29T11:21:41.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 10 monclient: _renew_subs 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa00aa740 con 0x7f9fa00b1f60 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 10 monclient: _renew_subs 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa00aabd0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.681+0000 7f9fa8484640 1 librados: init done 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.682+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9f98003d60 con 0x7f9fa00b1f60 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.682+0000 7f9f96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.682+0000 7f9f96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.682+0000 7f9f96ffd640 10 monclient: dump: 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:41.666 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.682+0000 7f9fa8484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.683+0000 7f9fa8484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.683+0000 7f9fa8484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.683+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa0075f10 con 0x7f9fa00b1f60 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.684+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f9f98003ec0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.685+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f9f980549c0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.685+0000 7f9f96ffd640 10 client.4346.objecter ms_dispatch 0x7f9fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:41.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.685+0000 7f9f96ffd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:41.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.685+0000 7f9f96ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:41.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.685+0000 7f9f96ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:21:41.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.686+0000 7f9fa5a1f640 10 monclient: get_auth_request con 0x7f9f8406df10 auth_method 0 2024-03-29T11:21:41.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.686+0000 7f9f96ffd640 10 client.4346.objecter ms_handle_connect 0x7f9f8406df10 2024-03-29T11:21:41.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.692+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f98050c60 con 0x7f9fa00b1f60 2024-03-29T11:21:41.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.692+0000 7f9f96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:41.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:41.692+0000 7f9f96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:41.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.019+0000 7f9fa8484640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.019+0000 7f9fa8484640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.019+0000 7f9fa8484640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:21:41.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.019+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa00aabd0 con 0x7f9fa00b1f60 2024-03-29T11:21:41.997 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:41.998 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:41.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.021+0000 7f9f96ffd640 1 -- 172.21.15.117:0/4052997736 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f9f98050e00 con 0x7f9fa00b1f60 2024-03-29T11:21:41.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.021+0000 7f9f96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:41.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.021+0000 7f9f96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:42.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.024+0000 7f9fa8484640 10 librados: watch_flush enter 2024-03-29T11:21:42.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.024+0000 7f9fa8484640 10 librados: watch_flush exit 2024-03-29T11:21:42.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:21:42.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:21:42.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 >> v1:172.21.15.117:6816/4276038175 conn(0x7f9f8406df10 legacy=0x7f9f840703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 10 monclient: shutdown 2024-03-29T11:21:42.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.025+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 >> v1:172.21.15.117:6790/0 conn(0x7f9fa00b1f60 legacy=0x7f9fa0145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.027+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 shutdown_connections 2024-03-29T11:21:42.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.027+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 >> 172.21.15.117:0/4052997736 conn(0x7f9fa0066a00 msgr2=0x7f9fa0142b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.027+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 shutdown_connections 2024-03-29T11:21:42.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.027+0000 7f9fa8484640 1 -- 172.21.15.117:0/4052997736 wait complete. 2024-03-29T11:21:42.003 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:42.027+0000 7f9fa8484640 1 librados: shutdown 2024-03-29T11:21:42.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:48.826 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:48.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.002+0000 7f8b2abe5640 10 monclient: get_monmap_and_config 2024-03-29T11:21:48.982 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.002+0000 7f8b2abe5640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.002+0000 7f8b2abe5640 10 monclient: monmap: 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:49.003365+0000 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:49.003365+0000 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.004+0000 7f8b2abe5640 1 Processor -- start 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 1 -- start start 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient: init 2024-03-29T11:21:48.983 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-b con 0x7f8b1c029f00 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-c con 0x7f8b1c02b090 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-a con 0x7f8b1c0e2c40 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.005+0000 7f8b2abe5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c02c0f0 con 0x7f8b1c0e2c40 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.006+0000 7f8b2abe5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c02a430 con 0x7f8b1c02b090 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.006+0000 7f8b2abe5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c02a5c0 con 0x7f8b1c029f00 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.006+0000 7f8b2abe5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:48.984 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.006+0000 7f8b2abe5640 10 monclient(hunting): authenticate will time out at 977.315028s 2024-03-29T11:21:48.985 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.009+0000 7f8b293e2640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f8b1c02b090 0x7f8b1c0e2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:46713/0 (socket says 172.21.15.117:46713) 2024-03-29T11:21:48.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.009+0000 7f8b293e2640 1 -- 172.21.15.117:0/3947278525 learned_addr learned my addr 172.21.15.117:0/3947278525 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:48.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202596668 0 0) 0x7f8b1c02a5c0 con 0x7f8b1c029f00 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b10003520 con 0x7f8b1c029f00 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980487834 0 0) 0x7f8b1c02a430 con 0x7f8b1c02b090 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _negotiate 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.011+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c02a5c0 con 0x7f8b1c02b090 2024-03-29T11:21:48.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809798873 0 0) 0x7f8b1c02a5c0 con 0x7f8b1c02b090 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 >> v1:172.21.15.117:6790/0 conn(0x7f8b1c029f00 legacy=0x7f8b1c0e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 >> v1:172.21.15.117:6789/0 conn(0x7f8b1c0e2c40 legacy=0x7f8b1c0f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: _finish_hunting 0 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 monclient: found mon.noname-c 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c006940 con 0x7f8b1c02b090 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: _finish_auth 0 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: _check_auth_tickets 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8b140012f0 con 0x7f8b1c02b090 2024-03-29T11:21:48.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b2abe5640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b2abe5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b28be1640 10 monclient: dump: 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:48.995 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.013+0000 7f8b2abe5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b140014c0 con 0x7f8b1c02b090 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b2abe5640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b2abe5640 4 set_mon_vals no callback set 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 1 -- 172.21.15.117:0/3947278525 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8b14001910 con 0x7f8b1c02b090 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.015+0000 7f8b28be1640 10 monclient: dump: 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:48.996 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 10 monclient: shutdown 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 1 -- 172.21.15.117:0/3947278525 >> v1:172.21.15.173:6789/0 conn(0x7f8b1c02b090 legacy=0x7f8b1c0e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 1 -- 172.21.15.117:0/3947278525 shutdown_connections 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 1 -- 172.21.15.117:0/3947278525 >> 172.21.15.117:0/3947278525 conn(0x7f8b1c0058f0 msgr2=0x7f8b1c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 1 -- 172.21.15.117:0/3947278525 shutdown_connections 2024-03-29T11:21:48.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.016+0000 7f8b2abe5640 1 -- 172.21.15.117:0/3947278525 wait complete. 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 10 monclient: build_initial_monmap 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 10 monclient: monmap: 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:48.998 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 librados: starting msgr at 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 librados: starting objecter 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 Processor -- start 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 -- start start 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 librados: setting wanted keys 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 1 librados: calling monclient init 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.017+0000 7f8b2abe5640 10 monclient: init 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-a con 0x7f8b1c029f00 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-b con 0x7f8b1c02b090 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient(hunting): picked mon.noname-c con 0x7f8b1c0e2c40 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:48.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c0dec70 con 0x7f8b1c029f00 2024-03-29T11:21:49.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c0df470 con 0x7f8b1c02b090 2024-03-29T11:21:49.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1c0df750 con 0x7f8b1c0e2c40 2024-03-29T11:21:49.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:49.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.018+0000 7f8b2abe5640 10 monclient(hunting): authenticate will time out at 977.327657s 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b29be3640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8b1c029f00 0x7f8b1c0d1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:60659/0 (socket says 172.21.15.117:60659) 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b29be3640 1 -- 172.21.15.117:0/734612682 learned_addr learned my addr 172.21.15.117:0/734612682 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f8b1c029f00 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f8b1c02b090 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390881294 0 0) 0x7f8b1c0dec70 con 0x7f8b1c029f00 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b08004eb0 con 0x7f8b1c029f00 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168980939 0 0) 0x7f8b1c0df470 con 0x7f8b1c02b090 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:21:49.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.024+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c0dec70 con 0x7f8b1c02b090 2024-03-29T11:21:49.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491746795 0 0) 0x7f8b1c0dec70 con 0x7f8b1c02b090 2024-03-29T11:21:49.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b1c0df470 con 0x7f8b1c02b090 2024-03-29T11:21:49.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8b140047c0 con 0x7f8b1c02b090 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:49.009 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:49.010 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:49.010 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.010 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:49.010 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:49.013 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:49.013 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:49.013 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:49.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.025+0000 7f8b1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:49.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f8b1c0e2c40 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116349194 0 0) 0x7f8b1c0df750 con 0x7f8b1c0e2c40 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.026+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c0dec70 con 0x7f8b1c0e2c40 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3251318245 0 0) 0x7f8b1c0df470 con 0x7f8b1c02b090 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 >> v1:172.21.15.117:6790/0 conn(0x7f8b1c0e2c40 legacy=0x7f8b1c185960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 >> v1:172.21.15.117:6789/0 conn(0x7f8b1c029f00 legacy=0x7f8b1c0d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 monclient: found mon.b 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:49.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c0de750 con 0x7f8b1c02b090 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 10 monclient: _renew_subs 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1c0dd8b0 con 0x7f8b1c02b090 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 10 monclient: _renew_subs 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c0d2580 con 0x7f8b1c02b090 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b2abe5640 1 librados: init done 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b140018e0 con 0x7f8b1c02b090 2024-03-29T11:21:49.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8b14004dd0 con 0x7f8b1c02b090 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.029+0000 7f8b1a7fc640 10 monclient: dump: 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:49.016 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:49.017 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:49.017 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:49.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.030+0000 7f8b1bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.031+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f8b1401ada0 con 0x7f8b1c02b090 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f8b14084a60 con 0x7f8b1c02b090 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b1a7fc640 10 client.4342.objecter ms_dispatch 0x7f8b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b1a7fc640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b1a7fc640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b1a7fc640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.032+0000 7f8b29be3640 10 monclient: get_auth_request con 0x7f8b0806d960 auth_method 0 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.033+0000 7f8b2abe5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.033+0000 7f8b2abe5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.033+0000 7f8b2abe5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.033+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c005080 con 0x7f8b1c02b090 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.033+0000 7f8b1a7fc640 10 client.4342.objecter ms_handle_connect 0x7f8b0806d960 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.041+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b14050340 con 0x7f8b1c02b090 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.041+0000 7f8b1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:49.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.041+0000 7f8b1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:49.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.364+0000 7f8b2abe5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.364+0000 7f8b2abe5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.364+0000 7f8b2abe5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:21:49.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.364+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b0c005330 con 0x7f8b1c02b090 2024-03-29T11:21:49.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.369+0000 7f8b1a7fc640 1 -- 172.21.15.117:0/734612682 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f8b14052270 con 0x7f8b1c02b090 2024-03-29T11:21:49.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.370+0000 7f8b1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:49.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.370+0000 7f8b1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:49.349 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:49.349 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:49.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 10 librados: watch_flush enter 2024-03-29T11:21:49.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 10 librados: watch_flush exit 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 >> v1:172.21.15.117:6816/4276038175 conn(0x7f8b0806d960 legacy=0x7f8b0806fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 10 monclient: shutdown 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:49.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.373+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 >> v1:172.21.15.173:6789/0 conn(0x7f8b1c02b090 legacy=0x7f8b1c036e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:49.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.376+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 shutdown_connections 2024-03-29T11:21:49.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.376+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 >> 172.21.15.117:0/734612682 conn(0x7f8b1c0058f0 msgr2=0x7f8b1c0e2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:49.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.376+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 shutdown_connections 2024-03-29T11:21:49.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.376+0000 7f8b2abe5640 1 -- 172.21.15.117:0/734612682 wait complete. 2024-03-29T11:21:49.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:49.376+0000 7f8b2abe5640 1 librados: shutdown 2024-03-29T11:21:49.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:56.176 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.355+0000 7f1b4c24d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.355+0000 7f1b4c24d640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.355+0000 7f1b4c24d640 10 monclient: monmap: 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:21:56.356485+0000 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:21:56.356485+0000 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:21:56.332 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:21:56.333 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.356+0000 7f1b4c24d640 1 Processor -- start 2024-03-29T11:21:56.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 1 -- start start 2024-03-29T11:21:56.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient: init 2024-03-29T11:21:56.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440b4330 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-b con 0x7f1b441470b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b441439b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b4406f2a0 con 0x7f1b441439b0 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44146b90 con 0x7f1b440b4330 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b2ca0 con 0x7f1b441470b0 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.357+0000 7f1b4c24d640 10 monclient(hunting): authenticate will time out at 984.666996s 2024-03-29T11:21:56.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.361+0000 7f1b4a7ea640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b441439b0 0x7f1b44151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58883/0 (socket says 172.21.15.117:58883) 2024-03-29T11:21:56.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.361+0000 7f1b4a7ea640 1 -- 172.21.15.117:0/1095283949 learned_addr learned my addr 172.21.15.117:0/1095283949 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:56.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898986933 0 0) 0x7f1b44146b90 con 0x7f1b440b4330 2024-03-29T11:21:56.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:21:56.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.363+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003520 con 0x7f1b440b4330 2024-03-29T11:21:56.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475700351 0 0) 0x7f1b30003520 con 0x7f1b440b4330 2024-03-29T11:21:56.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 >> v1:172.21.15.117:6790/0 conn(0x7f1b441470b0 legacy=0x7f1b440b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 >> v1:172.21.15.117:6789/0 conn(0x7f1b441439b0 legacy=0x7f1b44151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 monclient: found mon.noname-c 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4406e0c0 con 0x7f1b440b4330 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1b3c0019f0 con 0x7f1b440b4330 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.364+0000 7f1b48fe7640 10 monclient: dump: 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b4c24d640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b4c24d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b3c001c50 con 0x7f1b440b4330 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 1 -- 172.21.15.117:0/1095283949 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1b3c0020a0 con 0x7f1b440b4330 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b48fe7640 10 monclient: dump: 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.344 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:56.345 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:56.345 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:56.345 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b4c24d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:56.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.365+0000 7f1b4c24d640 4 set_mon_vals no callback set 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 10 monclient: shutdown 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 1 -- 172.21.15.117:0/1095283949 >> v1:172.21.15.173:6789/0 conn(0x7f1b440b4330 legacy=0x7f1b440b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 1 -- 172.21.15.117:0/1095283949 shutdown_connections 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 1 -- 172.21.15.117:0/1095283949 >> 172.21.15.117:0/1095283949 conn(0x7f1b44066a00 msgr2=0x7f1b44066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 1 -- 172.21.15.117:0/1095283949 shutdown_connections 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.366+0000 7f1b4c24d640 1 -- 172.21.15.117:0/1095283949 wait complete. 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.367+0000 7f1b4c24d640 10 monclient: build_initial_monmap 2024-03-29T11:21:56.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.367+0000 7f1b4c24d640 10 monclient: monmap: 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 librados: starting msgr at 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 librados: starting objecter 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 Processor -- start 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 -- start start 2024-03-29T11:21:56.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 librados: setting wanted keys 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 1 librados: calling monclient init 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.368+0000 7f1b4c24d640 10 monclient: init 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b441439b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-b con 0x7f1b441470b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b441dc060 addr v1:172.21.15.117:6790/0 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b441db4c0 con 0x7f1b441439b0 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44122730 con 0x7f1b441470b0 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44127680 con 0x7f1b441dc060 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:56.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.369+0000 7f1b4c24d640 10 monclient(hunting): authenticate will time out at 984.678218s 2024-03-29T11:21:56.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.375+0000 7f1b49fe9640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b441439b0 0x7f1b44125290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59039/0 (socket says 172.21.15.117:59039) 2024-03-29T11:21:56.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.375+0000 7f1b49fe9640 1 -- 172.21.15.117:0/2064269918 learned_addr learned my addr 172.21.15.117:0/2064269918 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:21:56.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.375+0000 7f1b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f1b441439b0 2024-03-29T11:21:56.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.375+0000 7f1b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f1b441dc060 2024-03-29T11:21:56.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434650191 0 0) 0x7f1b441db4c0 con 0x7f1b441439b0 2024-03-29T11:21:56.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:21:56.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.376+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28004eb0 con 0x7f1b441439b0 2024-03-29T11:21:56.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 client.?.objecter ms_handle_connect 0x7f1b441470b0 2024-03-29T11:21:56.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:56.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693577871 0 0) 0x7f1b44127680 con 0x7f1b441dc060 2024-03-29T11:21:56.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b441db4c0 con 0x7f1b441dc060 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123016302 0 0) 0x7f1b44122730 con 0x7f1b441470b0 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.378+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b44127680 con 0x7f1b441470b0 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475924941 0 0) 0x7f1b441db4c0 con 0x7f1b441dc060 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b44122730 con 0x7f1b441dc060 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1b340014d0 con 0x7f1b441dc060 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:56.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2739682673 0 0) 0x7f1b28004eb0 con 0x7f1b441439b0 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28005b20 con 0x7f1b441439b0 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1b3c0019b0 con 0x7f1b441439b0 2024-03-29T11:21:56.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.379+0000 7f1b3a7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2889228496 0 0) 0x7f1b28005b20 con 0x7f1b441439b0 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 >> v1:172.21.15.117:6790/0 conn(0x7f1b441dc060 legacy=0x7f1b441410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 >> v1:172.21.15.173:6789/0 conn(0x7f1b441470b0 legacy=0x7f1b441db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 1 monclient: found mon.a 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44141800 con 0x7f1b441439b0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 10 monclient: _renew_subs 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44141a50 con 0x7f1b441439b0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 10 monclient: _renew_subs 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b44126160 con 0x7f1b441439b0 2024-03-29T11:21:56.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.380+0000 7f1b4c24d640 1 librados: init done 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b3c003d60 con 0x7f1b441439b0 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1b3c0019b0 con 0x7f1b441439b0 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3a7fc640 10 monclient: dump: 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:21:56.366 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:21:56.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:21:56.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.383+0000 7f1b3bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.386+0000 7f1b4c24d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.386+0000 7f1b4c24d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.386+0000 7f1b4c24d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.386+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c005080 con 0x7f1b441439b0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.386+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f1b3c003f50 con 0x7f1b441439b0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.387+0000 7f1b497e8640 10 monclient: get_auth_request con 0x7f1b2806e540 auth_method 0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 10 client.4329.objecter ms_handle_connect 0x7f1b2806e540 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f1b3c0549e0 con 0x7f1b441439b0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 10 client.4329.objecter ms_dispatch 0x7f1b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:56.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.388+0000 7f1b3a7fc640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:21:56.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.397+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b3c050c80 con 0x7f1b441439b0 2024-03-29T11:21:56.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.397+0000 7f1b3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:56.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.397+0000 7f1b3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:56.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.732+0000 7f1b4c24d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.732+0000 7f1b4c24d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.732+0000 7f1b4c24d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:21:56.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.732+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b2c005330 con 0x7f1b441439b0 2024-03-29T11:21:56.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.733+0000 7f1b3a7fc640 1 -- 172.21.15.117:0/2064269918 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f1b3c050c80 con 0x7f1b441439b0 2024-03-29T11:21:56.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.733+0000 7f1b3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:56.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.733+0000 7f1b3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:56.713 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:21:56.713 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:56.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.738+0000 7f1b4c24d640 10 librados: watch_flush enter 2024-03-29T11:21:56.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.739+0000 7f1b4c24d640 10 librados: watch_flush exit 2024-03-29T11:21:56.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.739+0000 7f1b4c24d640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:21:56.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.739+0000 7f1b4c24d640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:21:56.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 >> v1:172.21.15.117:6816/4276038175 conn(0x7f1b2806e540 legacy=0x7f1b28070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 10 monclient: shutdown 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 >> v1:172.21.15.117:6789/0 conn(0x7f1b441439b0 legacy=0x7f1b44125290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 shutdown_connections 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 >> 172.21.15.117:0/2064269918 conn(0x7f1b44066a00 msgr2=0x7f1b440b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 shutdown_connections 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 -- 172.21.15.117:0/2064269918 wait complete. 2024-03-29T11:21:56.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:21:56.740+0000 7f1b4c24d640 1 librados: shutdown 2024-03-29T11:21:56.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:03.545 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:03.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.709+0000 7f8951a6e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.709+0000 7f8951a6e640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.709+0000 7f8951a6e640 10 monclient: monmap: 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:03.710355+0000 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:03.710355+0000 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:03.686 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.711+0000 7f8951a6e640 1 Processor -- start 2024-03-29T11:22:03.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.714+0000 7f8951a6e640 1 -- start start 2024-03-29T11:22:03.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.714+0000 7f8951a6e640 10 monclient: init 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.714+0000 7f8951a6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.714+0000 7f8951a6e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-c con 0x7f894c1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f894c13f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-a con 0x7f894c0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c0aaef0 con 0x7f894c0a9d70 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c145bd0 con 0x7f894c1457f0 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c145eb0 con 0x7f894c13f650 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.715+0000 7f8951a6e640 10 monclient(hunting): authenticate will time out at 992.024451s 2024-03-29T11:22:03.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.718+0000 7f894b7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f894c1457f0 0x7f894c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:57035/0 (socket says 172.21.15.117:57035) 2024-03-29T11:22:03.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.718+0000 7f894b7fe640 1 -- 172.21.15.117:0/3749033856 learned_addr learned my addr 172.21.15.117:0/3749033856 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401386502 0 0) 0x7f894c0aaef0 con 0x7f894c0a9d70 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.719+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8938003520 con 0x7f894c0a9d70 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658349782 0 0) 0x7f8938003520 con 0x7f894c0a9d70 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 >> v1:172.21.15.117:6790/0 conn(0x7f894c13f650 legacy=0x7f894c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 >> v1:172.21.15.173:6789/0 conn(0x7f894c1457f0 legacy=0x7f894c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 monclient: found mon.noname-a 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c146340 con 0x7f894c0a9d70 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8944001a10 con 0x7f894c0a9d70 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: dump: 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.700 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8944001be0 con 0x7f894c0a9d70 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 1 -- 172.21.15.117:0/3749033856 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8944001ff0 con 0x7f894c0a9d70 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.720+0000 7f894affd640 10 monclient: dump: 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.701 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 4 set_mon_vals no callback set 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 10 monclient: shutdown 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.721+0000 7f8951a6e640 1 -- 172.21.15.117:0/3749033856 >> v1:172.21.15.117:6789/0 conn(0x7f894c0a9d70 legacy=0x7f894c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.722+0000 7f8951a6e640 1 -- 172.21.15.117:0/3749033856 shutdown_connections 2024-03-29T11:22:03.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.722+0000 7f8951a6e640 1 -- 172.21.15.117:0/3749033856 >> 172.21.15.117:0/3749033856 conn(0x7f894c066a00 msgr2=0x7f894c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:03.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.722+0000 7f8951a6e640 1 -- 172.21.15.117:0/3749033856 shutdown_connections 2024-03-29T11:22:03.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.724+0000 7f8951a6e640 1 -- 172.21.15.117:0/3749033856 wait complete. 2024-03-29T11:22:03.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.724+0000 7f8951a6e640 10 monclient: build_initial_monmap 2024-03-29T11:22:03.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 10 monclient: monmap: 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 librados: starting msgr at 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 librados: starting objecter 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 Processor -- start 2024-03-29T11:22:03.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 -- start start 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 librados: setting wanted keys 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 1 librados: calling monclient init 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.725+0000 7f8951a6e640 10 monclient: init 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.726+0000 7f8951a6e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.726+0000 7f8951a6e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:03.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.726+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-a con 0x7f894c0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f894c13f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 10 monclient(hunting): picked mon.noname-c con 0x7f894c1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c142030 con 0x7f894c0a9d70 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c1422a0 con 0x7f894c13f650 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c135af0 con 0x7f894c1457f0 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:03.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.731+0000 7f8951a6e640 10 monclient(hunting): authenticate will time out at 992.040670s 2024-03-29T11:22:03.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f894b7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f894c0a9d70 0x7f894c139260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47601/0 (socket says 172.21.15.117:47601) 2024-03-29T11:22:03.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f894b7fe640 1 -- 172.21.15.117:0/3469697787 learned_addr learned my addr 172.21.15.117:0/3469697787 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter ms_handle_connect 0x7f894c0a9d70 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter ms_handle_connect 0x7f894c1457f0 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter ms_handle_connect 0x7f894c13f650 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:03.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.736+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254538180 0 0) 0x7f894c142030 con 0x7f894c0a9d70 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c004eb0 con 0x7f894c0a9d70 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581774786 0 0) 0x7f894c1422a0 con 0x7f894c13f650 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c142030 con 0x7f894c13f650 2024-03-29T11:22:03.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665700974 0 0) 0x7f894c135af0 con 0x7f894c1457f0 2024-03-29T11:22:03.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:03.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:03.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:03.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:22:03.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c1422a0 con 0x7f894c1457f0 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86550913 0 0) 0x7f894c142030 con 0x7f894c13f650 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.737+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f894c135af0 con 0x7f894c13f650 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f89340016c0 con 0x7f894c13f650 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.716 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1789637578 0 0) 0x7f892c004eb0 con 0x7f894c0a9d70 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c006c20 con 0x7f894c0a9d70 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688043010 0 0) 0x7f894c1422a0 con 0x7f894c1457f0 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c008370 con 0x7f894c1457f0 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8944001ef0 con 0x7f894c1457f0 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.717 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f893c001810 con 0x7f894c0a9d70 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.738+0000 7f8948ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.719 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1388872394 0 0) 0x7f892c008370 con 0x7f894c1457f0 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 >> v1:172.21.15.173:6789/0 conn(0x7f894c13f650 legacy=0x7f894c139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 >> v1:172.21.15.117:6789/0 conn(0x7f894c0a9d70 legacy=0x7f894c139260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 monclient: found mon.c 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c140e50 con 0x7f894c1457f0 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:03.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8951a6e640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8951a6e640 10 monclient: _renew_subs 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8951a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894c13bb00 con 0x7f894c1457f0 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8944001360 con 0x7f894c1457f0 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8951a6e640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8944001ef0 con 0x7f894c1457f0 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f8948ff9640 10 monclient: dump: 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:03.723 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.741+0000 7f894a7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.742+0000 7f8951a6e640 10 monclient: _renew_subs 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.743+0000 7f8951a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.743+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c142a30 con 0x7f894c1457f0 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.743+0000 7f8951a6e640 1 librados: init done 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.743+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f8944019b30 con 0x7f894c1457f0 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8951a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8951a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8951a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c140fe0 con 0x7f894c1457f0 2024-03-29T11:22:03.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f894b7fe640 10 monclient: get_auth_request con 0x7f892c06eb30 auth_method 0 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f8944053770 con 0x7f894c1457f0 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8948ff9640 10 client.4361.objecter ms_dispatch 0x7f894c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8948ff9640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.744+0000 7f8948ff9640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.745+0000 7f8948ff9640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T11:22:03.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.745+0000 7f8948ff9640 10 client.4361.objecter ms_handle_connect 0x7f892c06eb30 2024-03-29T11:22:03.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.755+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f894404fa10 con 0x7f894c1457f0 2024-03-29T11:22:03.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.755+0000 7f8948ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:03.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:03.755+0000 7f8948ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:04.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.073+0000 7f8951a6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.073+0000 7f8951a6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.073+0000 7f8951a6e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.073+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f894c075f10 con 0x7f894c1457f0 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.074+0000 7f8948ff9640 1 -- 172.21.15.117:0/3469697787 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f894404fbb0 con 0x7f894c1457f0 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.074+0000 7f8948ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:04.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.074+0000 7f8948ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:04.051 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:04.051 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 10 librados: watch_flush enter 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 10 librados: watch_flush exit 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 10 client.4361.objecter successfully canceled tick 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 >> v1:172.21.15.117:6816/4276038175 conn(0x7f892c06eb30 legacy=0x7f892c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 10 monclient: shutdown 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.077+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 >> v1:172.21.15.117:6790/0 conn(0x7f894c1457f0 legacy=0x7f894c1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.078+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 shutdown_connections 2024-03-29T11:22:04.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.078+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 >> 172.21.15.117:0/3469697787 conn(0x7f894c066a00 msgr2=0x7f894c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.081+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 shutdown_connections 2024-03-29T11:22:04.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.082+0000 7f8951a6e640 1 -- 172.21.15.117:0/3469697787 wait complete. 2024-03-29T11:22:04.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:04.082+0000 7f8951a6e640 1 librados: shutdown 2024-03-29T11:22:04.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:10.885 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.064+0000 7fb20fd93640 10 monclient: get_monmap_and_config 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.064+0000 7fb20fd93640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.064+0000 7fb20fd93640 10 monclient: monmap: 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:11.065603+0000 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:11.065603+0000 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.041 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:11.042 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:11.042 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:11.042 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.068+0000 7fb20fd93640 1 Processor -- start 2024-03-29T11:22:11.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.069+0000 7fb20fd93640 1 -- start start 2024-03-29T11:22:11.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.069+0000 7fb20fd93640 10 monclient: init 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-c con 0x7fb2080b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-a con 0x7fb20814a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-b con 0x7fb2080ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080aba70 con 0x7fb20814a700 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080aecb0 con 0x7fb2080b0770 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080af150 con 0x7fb2080ac5a0 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.070+0000 7fb20fd93640 10 monclient(hunting): authenticate will time out at 999.379503s 2024-03-29T11:22:11.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.075+0000 7fb20d32e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb20814a700 0x7fb2080abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:46239/0 (socket says 172.21.15.117:46239) 2024-03-29T11:22:11.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.075+0000 7fb20d32e640 1 -- 172.21.15.117:0/1565213350 learned_addr learned my addr 172.21.15.117:0/1565213350 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.076+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639553328 0 0) 0x7fb2080af150 con 0x7fb2080ac5a0 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f4003520 con 0x7fb2080ac5a0 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486449393 0 0) 0x7fb2080aba70 con 0x7fb20814a700 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2080af150 con 0x7fb20814a700 2024-03-29T11:22:11.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491183681 0 0) 0x7fb1f4003520 con 0x7fb2080ac5a0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 >> v1:172.21.15.173:6789/0 conn(0x7fb2080b0770 legacy=0x7fb208149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 >> v1:172.21.15.117:6789/0 conn(0x7fb20814a700 legacy=0x7fb2080abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 monclient: found mon.noname-b 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080ac9d0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb2040015c0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20cb2d640 10 monclient: dump: 2024-03-29T11:22:11.055 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20fd93640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:22:11.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.077+0000 7fb20fd93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:11.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.081+0000 7fb20cb2d640 1 -- 172.21.15.117:0/1565213350 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2040017e0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.058 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.081+0000 7fb20cb2d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.081+0000 7fb20fd93640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.081+0000 7fb20fd93640 4 set_mon_vals no callback set 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.082+0000 7fb20fd93640 10 monclient: shutdown 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.082+0000 7fb20fd93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.082+0000 7fb20fd93640 1 -- 172.21.15.117:0/1565213350 >> v1:172.21.15.117:6790/0 conn(0x7fb2080ac5a0 legacy=0x7fb208151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.083+0000 7fb20fd93640 1 -- 172.21.15.117:0/1565213350 shutdown_connections 2024-03-29T11:22:11.059 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.083+0000 7fb20fd93640 1 -- 172.21.15.117:0/1565213350 >> 172.21.15.117:0/1565213350 conn(0x7fb20806b660 msgr2=0x7fb20806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.084+0000 7fb20fd93640 1 -- 172.21.15.117:0/1565213350 shutdown_connections 2024-03-29T11:22:11.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.086+0000 7fb20fd93640 1 -- 172.21.15.117:0/1565213350 wait complete. 2024-03-29T11:22:11.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.087+0000 7fb20fd93640 10 monclient: build_initial_monmap 2024-03-29T11:22:11.063 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.087+0000 7fb20fd93640 10 monclient: monmap: 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:11.064 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 librados: starting msgr at 2024-03-29T11:22:11.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 librados: starting objecter 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 Processor -- start 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 -- start start 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 librados: setting wanted keys 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 1 librados: calling monclient init 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.088+0000 7fb20fd93640 10 monclient: init 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.089+0000 7fb20fd93640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.089+0000 7fb20fd93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.089+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-c con 0x7fb2080ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:11.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.089+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-b con 0x7fb2080b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20db2f640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb2080ac5a0 0x7fb2081e3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47373/0 (socket says 172.21.15.117:47373) 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20e330640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb20814a700 0x7fb208147ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:46479/0 (socket says 172.21.15.117:46479) 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 10 monclient(hunting): picked mon.noname-a con 0x7fb20814a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208121e90 con 0x7fb20814a700 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2081226c0 con 0x7fb2080b0770 2024-03-29T11:22:11.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20811fa30 con 0x7fb2080ac5a0 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.092+0000 7fb20fd93640 10 monclient(hunting): authenticate will time out at 999.401858s 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb20db2f640 1 -- 172.21.15.117:0/160339911 learned_addr learned my addr 172.21.15.117:0/160339911 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb20814a700 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528263766 0 0) 0x7fb208121e90 con 0x7fb20814a700 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec004eb0 con 0x7fb20814a700 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb2080ac5a0 2024-03-29T11:22:11.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.093+0000 7fb1fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:11.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469558614 0 0) 0x7fb1ec004eb0 con 0x7fb20814a700 2024-03-29T11:22:11.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb208121e90 con 0x7fb20814a700 2024-03-29T11:22:11.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047515438 0 0) 0x7fb20811fa30 con 0x7fb2080ac5a0 2024-03-29T11:22:11.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:11.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec004eb0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb204003790 con 0x7fb20814a700 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.073 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.095+0000 7fb1fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3188047208 0 0) 0x7fb1ec004eb0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb20811fa30 con 0x7fb2080ac5a0 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb1f8003f40 con 0x7fb2080ac5a0 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.096+0000 7fb1fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.074 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:11.075 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1137348821 0 0) 0x7fb20811fa30 con 0x7fb2080ac5a0 2024-03-29T11:22:11.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 >> v1:172.21.15.173:6789/0 conn(0x7fb2080b0770 legacy=0x7fb2081204a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:22:11.077 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 >> v1:172.21.15.117:6789/0 conn(0x7fb20814a700 legacy=0x7fb208147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 1 monclient: found mon.c 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb208120bb0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb1fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 10 monclient: _renew_subs 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb208120e90 con 0x7fb2080ac5a0 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:11.078 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 10 monclient: _renew_subs 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2081487c0 con 0x7fb2080ac5a0 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.097+0000 7fb20fd93640 1 librados: init done 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.099+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f8004450 con 0x7fb2080ac5a0 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.099+0000 7fb1fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.099+0000 7fb1fffff640 4 set_mon_vals no callback set 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.099+0000 7fb20fd93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.100+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb1f8003f40 con 0x7fb2080ac5a0 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb20fd93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb20fd93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f0005080 con 0x7fb2080ac5a0 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb1fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb1fe7fc640 10 monclient: dump: 2024-03-29T11:22:11.080 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:11.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.103+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fb1f8000f50 con 0x7fb2080ac5a0 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fb1f8084710 con 0x7fb2080ac5a0 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb1fe7fc640 10 client.4367.objecter ms_dispatch 0x7fb208000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb1fe7fc640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb1fe7fc640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb1fe7fc640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.104+0000 7fb20d32e640 10 monclient: get_auth_request con 0x7fb1ec06e0f0 auth_method 0 2024-03-29T11:22:11.082 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.105+0000 7fb1fe7fc640 10 client.4367.objecter ms_handle_connect 0x7fb1ec06e0f0 2024-03-29T11:22:11.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.113+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1f804fe20 con 0x7fb2080ac5a0 2024-03-29T11:22:11.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.113+0000 7fb1fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:11.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.113+0000 7fb1fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:11.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.434+0000 7fb20fd93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.435+0000 7fb20fd93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.435+0000 7fb20fd93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:11.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.435+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1f0005330 con 0x7fb2080ac5a0 2024-03-29T11:22:11.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.436+0000 7fb1fe7fc640 1 -- 172.21.15.117:0/160339911 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fb1f804fe20 con 0x7fb2080ac5a0 2024-03-29T11:22:11.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.436+0000 7fb1fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:11.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.436+0000 7fb1fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:11.413 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:11.413 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:11.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 10 librados: watch_flush enter 2024-03-29T11:22:11.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 10 librados: watch_flush exit 2024-03-29T11:22:11.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:22:11.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 >> v1:172.21.15.117:6816/4276038175 conn(0x7fb1ec06e0f0 legacy=0x7fb1ec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 10 monclient: shutdown 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 >> v1:172.21.15.117:6790/0 conn(0x7fb2080ac5a0 legacy=0x7fb2081e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.440+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 shutdown_connections 2024-03-29T11:22:11.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.441+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 >> 172.21.15.117:0/160339911 conn(0x7fb20806b660 msgr2=0x7fb2080b3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:11.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.441+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 shutdown_connections 2024-03-29T11:22:11.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.441+0000 7fb20fd93640 1 -- 172.21.15.117:0/160339911 wait complete. 2024-03-29T11:22:11.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:11.441+0000 7fb20fd93640 1 librados: shutdown 2024-03-29T11:22:11.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:18.242 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.406+0000 7f405359e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.406+0000 7f405359e640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.406+0000 7f405359e640 10 monclient: monmap: 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:18.407895+0000 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:18.407895+0000 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.383 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.384 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:18.384 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:18.384 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:18.384 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.407+0000 7f405359e640 1 Processor -- start 2024-03-29T11:22:18.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.407+0000 7f405359e640 1 -- start start 2024-03-29T11:22:18.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.407+0000 7f405359e640 10 monclient: init 2024-03-29T11:22:18.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.408+0000 7f405359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.408+0000 7f405359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:18.386 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.408+0000 7f405359e640 10 monclient(hunting): picked mon.noname-c con 0x7f404c0389e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:18.386 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.408+0000 7f405359e640 10 monclient(hunting): picked mon.noname-a con 0x7f404c038dc0 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:18.386 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 10 monclient(hunting): picked mon.noname-b con 0x7f404c030660 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:18.386 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c0c54c0 con 0x7f404c038dc0 2024-03-29T11:22:18.386 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c036430 con 0x7f404c0389e0 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c039840 con 0x7f404c030660 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f405359e640 10 monclient(hunting): authenticate will time out at 1006.718399s 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f4052d9d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f404c030660 0x7f404c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51506/0 (socket says 172.21.15.117:51506) 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.409+0000 7f4052d9d640 1 -- 172.21.15.117:0/1808587760 learned_addr learned my addr 172.21.15.117:0/1808587760 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395119177 0 0) 0x7f404c036430 con 0x7f404c0389e0 2024-03-29T11:22:18.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403c003520 con 0x7f404c0389e0 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784716136 0 0) 0x7f404c039840 con 0x7f404c030660 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c036430 con 0x7f404c030660 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679295142 0 0) 0x7f403c003520 con 0x7f404c0389e0 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 >> v1:172.21.15.117:6790/0 conn(0x7f404c030660 legacy=0x7f404c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.410+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 >> v1:172.21.15.117:6789/0 conn(0x7f404c038dc0 legacy=0x7f404c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:18.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 1 monclient: found mon.noname-c 2024-03-29T11:22:18.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c035690 con 0x7f404c0389e0 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4044001690 con 0x7f404c0389e0 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: dump: 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:18.391 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405359e640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40440018f0 con 0x7f404c0389e0 2024-03-29T11:22:18.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 1 -- 172.21.15.117:0/1808587760 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4044001d40 con 0x7f404c0389e0 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405159a640 10 monclient: dump: 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:18.393 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.413+0000 7f405359e640 4 set_mon_vals no callback set 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 10 monclient: shutdown 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 1 -- 172.21.15.117:0/1808587760 >> v1:172.21.15.173:6789/0 conn(0x7f404c0389e0 legacy=0x7f404c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 1 -- 172.21.15.117:0/1808587760 shutdown_connections 2024-03-29T11:22:18.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 1 -- 172.21.15.117:0/1808587760 >> 172.21.15.117:0/1808587760 conn(0x7f404c005c00 msgr2=0x7f404c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.414+0000 7f405359e640 1 -- 172.21.15.117:0/1808587760 shutdown_connections 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.415+0000 7f405359e640 1 -- 172.21.15.117:0/1808587760 wait complete. 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 10 monclient: build_initial_monmap 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 10 monclient: monmap: 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 librados: starting msgr at 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 librados: starting objecter 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 Processor -- start 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 -- start start 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 librados: setting wanted keys 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 1 librados: calling monclient init 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.416+0000 7f405359e640 10 monclient: init 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.417+0000 7f405359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.417+0000 7f405359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f405359e640 10 monclient(hunting): picked mon.noname-b con 0x7f404c030660 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f405259c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f404c030660 0x7f404c0c24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:57742/0 (socket says 172.21.15.117:57742) 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f405259c640 1 -- 172.21.15.117:0/1195590489 learned_addr learned my addr 172.21.15.117:0/1195590489 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f4042ffd640 10 client.?.objecter ms_handle_connect 0x7f404c030660 2024-03-29T11:22:18.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f4042ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f405359e640 10 monclient(hunting): picked mon.noname-c con 0x7f404c0389e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f4042ffd640 10 client.?.objecter ms_handle_connect 0x7f404c0389e0 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f4042ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.418+0000 7f405359e640 10 monclient(hunting): picked mon.noname-a con 0x7f404c0c2bb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c02fa10 con 0x7f404c0c2bb0 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c02ed80 con 0x7f404c030660 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404c02ef10 con 0x7f404c0389e0 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f405359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f405359e640 10 monclient(hunting): authenticate will time out at 1006.728048s 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144483414 0 0) 0x7f404c02ed80 con 0x7f404c030660 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.419+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034004eb0 con 0x7f404c030660 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307816307 0 0) 0x7f404c02ef10 con 0x7f404c0389e0 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c02ed80 con 0x7f404c0389e0 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16018130 0 0) 0x7f4034004eb0 con 0x7f404c030660 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f404c02ef10 con 0x7f404c030660 2024-03-29T11:22:18.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4044001650 con 0x7f404c030660 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:18.405 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.420+0000 7f4042ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.421+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 307553953 0 0) 0x7f404c02ef10 con 0x7f404c030660 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.421+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 >> v1:172.21.15.117:6790/0 conn(0x7f404c0389e0 legacy=0x7f404c15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.421+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 >> v1:172.21.15.117:6789/0 conn(0x7f404c0c2bb0 legacy=0x7f404c1682e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 1 monclient: found mon.b 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c02f0a0 con 0x7f404c030660 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f405359e640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f405359e640 10 monclient: _renew_subs 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40440039c0 con 0x7f404c030660 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f405359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c0c7820 con 0x7f404c030660 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.422+0000 7f4050d99640 4 set_mon_vals no callback set 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f405359e640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:18.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f405359e640 10 monclient: _renew_subs 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f405359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c02f340 con 0x7f404c030660 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f405359e640 1 librados: init done 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4044003dd0 con 0x7f404c030660 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f4042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f4042ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.423+0000 7f4042ffd640 10 monclient: dump: 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:18.408 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:18.409 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:18.409 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:18.409 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:18.409 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:18.409 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f405359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f405359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f405359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c007b90 con 0x7f404c030660 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4044003f30 con 0x7f404c030660 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f4044084b20 con 0x7f404c030660 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.425+0000 7f4042ffd640 10 client.4360.objecter ms_dispatch 0x7f4054000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.426+0000 7f4042ffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:18.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.426+0000 7f4042ffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.426+0000 7f4042ffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.426+0000 7f4051d9b640 10 monclient: get_auth_request con 0x7f403406dc10 auth_method 0 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.427+0000 7f4042ffd640 10 client.4360.objecter ms_handle_connect 0x7f403406dc10 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.434+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4044050460 con 0x7f404c030660 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.434+0000 7f4042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:18.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.434+0000 7f4042ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:18.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.729+0000 7f405359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.729+0000 7f405359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.729+0000 7f405359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:18.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.729+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f404c02fc00 con 0x7f404c030660 2024-03-29T11:22:18.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.730+0000 7f4042ffd640 1 -- 172.21.15.117:0/1195590489 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f40440841d0 con 0x7f404c030660 2024-03-29T11:22:18.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.730+0000 7f4042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:18.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.730+0000 7f4042ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:18.707 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:18.707 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.733+0000 7f405359e640 10 librados: watch_flush enter 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 10 librados: watch_flush exit 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 >> v1:172.21.15.117:6816/4276038175 conn(0x7f403406dc10 legacy=0x7f40340700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 10 monclient: shutdown 2024-03-29T11:22:18.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 >> v1:172.21.15.173:6789/0 conn(0x7f404c030660 legacy=0x7f404c0c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 shutdown_connections 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 >> 172.21.15.117:0/1195590489 conn(0x7f404c005c00 msgr2=0x7f404c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 shutdown_connections 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 -- 172.21.15.117:0/1195590489 wait complete. 2024-03-29T11:22:18.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:18.734+0000 7f405359e640 1 librados: shutdown 2024-03-29T11:22:18.725 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:25.532 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.704+0000 7fd88377a640 10 monclient: get_monmap_and_config 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.704+0000 7fd88377a640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.704+0000 7fd88377a640 10 monclient: monmap: 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:25.705615+0000 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:25.705615+0000 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:25.681 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:25.682 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:25.682 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.705+0000 7fd88377a640 1 Processor -- start 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.705+0000 7fd88377a640 1 -- start start 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.705+0000 7fd88377a640 10 monclient: init 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-c con 0x7fd87c1422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c142690 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-a con 0x7fd87c0b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c142c80 con 0x7fd87c0b17d0 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c0aae40 con 0x7fd87c1422b0 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c13fbd0 con 0x7fd87c142690 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd88377a640 10 monclient(hunting): authenticate will time out at 1014.015611s 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd880d15640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd87c142690 0x7fd87c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58792/0 (socket says 172.21.15.117:58792) 2024-03-29T11:22:25.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.706+0000 7fd880d15640 1 -- 172.21.15.117:0/632417223 learned_addr learned my addr 172.21.15.117:0/632417223 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:25.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979767946 0 0) 0x7fd87c0aae40 con 0x7fd87c1422b0 2024-03-29T11:22:25.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd864003520 con 0x7fd87c1422b0 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698150449 0 0) 0x7fd87c13fbd0 con 0x7fd87c142690 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c0aae40 con 0x7fd87c142690 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441442945 0 0) 0x7fd87c142c80 con 0x7fd87c0b17d0 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.707+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c13fbd0 con 0x7fd87c0b17d0 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618838067 0 0) 0x7fd864003520 con 0x7fd87c1422b0 2024-03-29T11:22:25.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 >> v1:172.21.15.117:6790/0 conn(0x7fd87c142690 legacy=0x7fd87c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 >> v1:172.21.15.117:6789/0 conn(0x7fd87c0b17d0 legacy=0x7fd87c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 monclient: found mon.noname-c 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c151940 con 0x7fd87c1422b0 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd874001690 con 0x7fd87c1422b0 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: dump: 2024-03-29T11:22:25.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd88377a640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:22:25.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd88377a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8740017f0 con 0x7fd87c1422b0 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd87bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd88377a640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.708+0000 7fd88377a640 4 set_mon_vals no callback set 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd87bfff640 1 -- 172.21.15.117:0/632417223 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd874001b80 con 0x7fd87c1422b0 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd87bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd87bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd87bfff640 10 monclient: dump: 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.689 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd88377a640 10 monclient: shutdown 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd88377a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.709+0000 7fd88377a640 1 -- 172.21.15.117:0/632417223 >> v1:172.21.15.173:6789/0 conn(0x7fd87c1422b0 legacy=0x7fd87c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.710+0000 7fd88377a640 1 -- 172.21.15.117:0/632417223 shutdown_connections 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.710+0000 7fd88377a640 1 -- 172.21.15.117:0/632417223 >> 172.21.15.117:0/632417223 conn(0x7fd87c066a00 msgr2=0x7fd87c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.710+0000 7fd88377a640 1 -- 172.21.15.117:0/632417223 shutdown_connections 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.710+0000 7fd88377a640 1 -- 172.21.15.117:0/632417223 wait complete. 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 10 monclient: build_initial_monmap 2024-03-29T11:22:25.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 10 monclient: monmap: 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 librados: starting msgr at 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 librados: starting objecter 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 Processor -- start 2024-03-29T11:22:25.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 -- start start 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 librados: setting wanted keys 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 1 librados: calling monclient init 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.711+0000 7fd88377a640 10 monclient: init 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-c con 0x7fd87c0b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c1422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient(hunting): picked mon.noname-a con 0x7fd87c142690 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c13a730 con 0x7fd87c142690 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c1db9e0 con 0x7fd87c1422b0 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87c0b0640 con 0x7fd87c0b17d0 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.712+0000 7fd88377a640 10 monclient(hunting): authenticate will time out at 1014.021903s 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd881516640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd87c0b17d0 0x7fd87c1db2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58958/0 (socket says 172.21.15.117:58958) 2024-03-29T11:22:25.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd881516640 1 -- 172.21.15.117:0/2079040484 learned_addr learned my addr 172.21.15.117:0/2079040484 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:25.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter ms_handle_connect 0x7fd87c0b17d0 2024-03-29T11:22:25.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter ms_handle_connect 0x7fd87c1422b0 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter ms_handle_connect 0x7fd87c142690 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283388659 0 0) 0x7fd87c1db9e0 con 0x7fd87c1422b0 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:22:25.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.713+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868004eb0 con 0x7fd87c1422b0 2024-03-29T11:22:25.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459985822 0 0) 0x7fd87c13a730 con 0x7fd87c142690 2024-03-29T11:22:25.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c1db9e0 con 0x7fd87c142690 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798763437 0 0) 0x7fd868004eb0 con 0x7fd87c1422b0 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd87c13a730 con 0x7fd87c1422b0 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd86c001ab0 con 0x7fd87c1422b0 2024-03-29T11:22:25.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.702 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:25.703 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122116641 0 0) 0x7fd87c1db9e0 con 0x7fd87c142690 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd868004eb0 con 0x7fd87c142690 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd8700017b0 con 0x7fd87c142690 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.705 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307647209 0 0) 0x7fd87c0b0640 con 0x7fd87c0b17d0 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c1db9e0 con 0x7fd87c0b17d0 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1900682810 0 0) 0x7fd87c13a730 con 0x7fd87c1422b0 2024-03-29T11:22:25.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 >> v1:172.21.15.117:6790/0 conn(0x7fd87c0b17d0 legacy=0x7fd87c1db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.714+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 >> v1:172.21.15.117:6789/0 conn(0x7fd87c142690 legacy=0x7fd87c13a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 1 monclient: found mon.b 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:25.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c1dadb0 con 0x7fd87c1422b0 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 10 monclient: _renew_subs 2024-03-29T11:22:25.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c14e640 con 0x7fd87c1422b0 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 10 monclient: _renew_subs 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c14dd80 con 0x7fd87c1422b0 2024-03-29T11:22:25.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.715+0000 7fd88377a640 1 librados: init done 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd86c001fa0 con 0x7fd87c1422b0 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd86c004d00 con 0x7fd87c1422b0 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd87affd640 4 set_mon_vals no callback set 2024-03-29T11:22:25.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.716+0000 7fd879ffb640 10 monclient: dump: 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:25.712 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.718+0000 7fd88377a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.718+0000 7fd88377a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.718+0000 7fd88377a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:25.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.718+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c005080 con 0x7fd87c1422b0 2024-03-29T11:22:25.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.718+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd86c01acd0 con 0x7fd87c1422b0 2024-03-29T11:22:25.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fd86c084b90 con 0x7fd87c1422b0 2024-03-29T11:22:25.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd879ffb640 10 client.4366.objecter ms_dispatch 0x7fd87c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:25.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd879ffb640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:25.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd879ffb640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:25.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd879ffb640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:22:25.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.719+0000 7fd881516640 10 monclient: get_auth_request con 0x7fd86806e010 auth_method 0 2024-03-29T11:22:25.715 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.720+0000 7fd879ffb640 10 client.4366.objecter ms_handle_connect 0x7fd86806e010 2024-03-29T11:22:25.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.725+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd86c050470 con 0x7fd87c1422b0 2024-03-29T11:22:25.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.726+0000 7fd879ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:25.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:25.726+0000 7fd879ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:26.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.091+0000 7fd88377a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.091+0000 7fd88377a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.091+0000 7fd88377a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:26.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.091+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd85c005330 con 0x7fd87c1422b0 2024-03-29T11:22:26.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.092+0000 7fd879ffb640 1 -- 172.21.15.117:0/2079040484 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fd86c0523a0 con 0x7fd87c1422b0 2024-03-29T11:22:26.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.092+0000 7fd879ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:26.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.092+0000 7fd879ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:26.069 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:26.069 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:26.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 10 librados: watch_flush enter 2024-03-29T11:22:26.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 10 librados: watch_flush exit 2024-03-29T11:22:26.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:22:26.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd86806e010 legacy=0x7fd8680704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 10 monclient: shutdown 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.096+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 >> v1:172.21.15.173:6789/0 conn(0x7fd87c1422b0 legacy=0x7fd87c139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.097+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 shutdown_connections 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.097+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 >> 172.21.15.117:0/2079040484 conn(0x7fd87c066a00 msgr2=0x7fd87c13ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.097+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 shutdown_connections 2024-03-29T11:22:26.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.097+0000 7fd88377a640 1 -- 172.21.15.117:0/2079040484 wait complete. 2024-03-29T11:22:26.074 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:26.097+0000 7fd88377a640 1 librados: shutdown 2024-03-29T11:22:26.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:32.897 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:33.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.051+0000 7f58f1a0d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.051+0000 7f58f1a0d640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.051+0000 7f58f1a0d640 10 monclient: monmap: 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:33.052292+0000 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:33.052292+0000 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.028 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:33.029 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:33.029 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:33.029 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.052+0000 7f58f1a0d640 1 Processor -- start 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.052+0000 7f58f1a0d640 1 -- start start 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.052+0000 7f58f1a0d640 10 monclient: init 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f58ec06c250 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f58ec072400 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f58ec06c630 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec06d760 con 0x7f58ec06c630 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec070500 con 0x7f58ec072400 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0aff50 con 0x7f58ec06c250 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58f1a0d640 10 monclient(hunting): authenticate will time out at 1021.362245s 2024-03-29T11:22:33.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58eaffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f58ec06c250 0x7f58ec071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39138/0 (socket says 172.21.15.117:39138) 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58eaffd640 1 -- 172.21.15.117:0/2095200262 learned_addr learned my addr 172.21.15.117:0/2095200262 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.053+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654912943 0 0) 0x7f58ec0aff50 con 0x7f58ec06c250 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d4003520 con 0x7f58ec06c250 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400054009 0 0) 0x7f58d4003520 con 0x7f58ec06c250 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 >> v1:172.21.15.173:6789/0 conn(0x7f58ec072400 legacy=0x7f58ec1465b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:33.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.054+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 >> v1:172.21.15.117:6789/0 conn(0x7f58ec06c630 legacy=0x7f58ec151ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:33.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:33.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0b00e0 con 0x7f58ec06c250 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f58e0001690 con 0x7f58ec06c250 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58e9ffb640 10 monclient: dump: 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.034 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.055+0000 7f58f1a0d640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58f1a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e00018f0 con 0x7f58ec06c250 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58e9ffb640 1 -- 172.21.15.117:0/2095200262 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f58e0001d40 con 0x7f58ec06c250 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58f1a0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.056+0000 7f58f1a0d640 4 set_mon_vals no callback set 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.057+0000 7f58e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.057+0000 7f58e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.057+0000 7f58e9ffb640 10 monclient: dump: 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:33.035 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:33.036 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.058+0000 7f58f1a0d640 10 monclient: shutdown 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.058+0000 7f58f1a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.058+0000 7f58f1a0d640 1 -- 172.21.15.117:0/2095200262 >> v1:172.21.15.117:6790/0 conn(0x7f58ec06c250 legacy=0x7f58ec071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.058+0000 7f58f1a0d640 1 -- 172.21.15.117:0/2095200262 shutdown_connections 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.058+0000 7f58f1a0d640 1 -- 172.21.15.117:0/2095200262 >> 172.21.15.117:0/2095200262 conn(0x7f58ec067610 msgr2=0x7f58ec0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.059+0000 7f58f1a0d640 1 -- 172.21.15.117:0/2095200262 shutdown_connections 2024-03-29T11:22:33.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.059+0000 7f58f1a0d640 1 -- 172.21.15.117:0/2095200262 wait complete. 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.061+0000 7f58f1a0d640 10 monclient: build_initial_monmap 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.061+0000 7f58f1a0d640 10 monclient: monmap: 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:33.038 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.061+0000 7f58f1a0d640 1 librados: starting msgr at 2024-03-29T11:22:33.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.061+0000 7f58f1a0d640 1 librados: starting objecter 2024-03-29T11:22:33.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.061+0000 7f58f1a0d640 1 Processor -- start 2024-03-29T11:22:33.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.062+0000 7f58f1a0d640 1 -- start start 2024-03-29T11:22:33.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.062+0000 7f58f1a0d640 1 librados: setting wanted keys 2024-03-29T11:22:33.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.062+0000 7f58f1a0d640 1 librados: calling monclient init 2024-03-29T11:22:33.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.062+0000 7f58f1a0d640 10 monclient: init 2024-03-29T11:22:33.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:33.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:33.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f58ec06c250 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:33.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f58ec06c630 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:33.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58eaffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f58ec06c250 0x7f58ec1df4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:44314/0 (socket says 172.21.15.117:44314) 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58eaffd640 1 -- 172.21.15.117:0/1642421406 learned_addr learned my addr 172.21.15.117:0/1642421406 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f58ec072400 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.064+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec1ded80 con 0x7f58ec06c630 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec1dfc00 con 0x7f58ec06c250 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 client.?.objecter ms_handle_connect 0x7f58ec06c250 2024-03-29T11:22:33.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 client.?.objecter ms_handle_connect 0x7f58ec072400 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0abba0 con 0x7f58ec072400 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58f1a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887422813 0 0) 0x7f58ec1dfc00 con 0x7f58ec06c250 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58f1a0d640 10 monclient(hunting): authenticate will time out at 1021.374497s 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8004eb0 con 0x7f58ec06c250 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839553912 0 0) 0x7f58ec0abba0 con 0x7f58ec072400 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec1dfc00 con 0x7f58ec072400 2024-03-29T11:22:33.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613621050 0 0) 0x7f58ec1dfc00 con 0x7f58ec072400 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58ec0abba0 con 0x7f58ec072400 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f58e4001a60 con 0x7f58ec072400 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): dump: 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:33.052 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.065+0000 7f58d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3878506556 0 0) 0x7f58ec0abba0 con 0x7f58ec072400 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 >> v1:172.21.15.173:6789/0 conn(0x7f58ec06c250 legacy=0x7f58ec1df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 >> v1:172.21.15.117:6789/0 conn(0x7f58ec06c630 legacy=0x7f58ec1d8950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 1 monclient: found mon.c 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec1defd0 con 0x7f58ec072400 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:33.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.066+0000 7f58d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58e4001ff0 con 0x7f58ec072400 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 10 monclient: _renew_subs 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec0af400 con 0x7f58ec072400 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f58e4003b10 con 0x7f58ec072400 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58d3fff640 10 monclient: dump: 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:33.054 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 10 monclient: _renew_subs 2024-03-29T11:22:33.055 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec0ad100 con 0x7f58ec072400 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58f1a0d640 1 librados: init done 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.067+0000 7f58e97fa640 4 set_mon_vals no callback set 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.069+0000 7f58f1a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.069+0000 7f58f1a0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.069+0000 7f58f1a0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.069+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec1df1c0 con 0x7f58ec072400 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.069+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f58e4019ae0 con 0x7f58ec072400 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f58e4083b10 con 0x7f58ec072400 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58d3fff640 10 client.4385.objecter ms_dispatch 0x7f58ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58d3fff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58d3fff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58d3fff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.070+0000 7f58eaffd640 10 monclient: get_auth_request con 0x7f58d806dd40 auth_method 0 2024-03-29T11:22:33.056 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.071+0000 7f58d3fff640 10 client.4385.objecter ms_handle_connect 0x7f58d806dd40 2024-03-29T11:22:33.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.077+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58e404f220 con 0x7f58ec072400 2024-03-29T11:22:33.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.077+0000 7f58d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:33.057 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.077+0000 7f58d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:33.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58f1a0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58f1a0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58f1a0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:33.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58ec075f10 con 0x7f58ec072400 2024-03-29T11:22:33.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58d3fff640 1 -- 172.21.15.117:0/1642421406 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f58e4051320 con 0x7f58ec072400 2024-03-29T11:22:33.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:33.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.367+0000 7f58d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:33.344 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:33.344 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 10 librados: watch_flush enter 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 10 librados: watch_flush exit 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 >> v1:172.21.15.117:6816/4276038175 conn(0x7f58d806dd40 legacy=0x7f58d8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 10 monclient: shutdown 2024-03-29T11:22:33.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 >> v1:172.21.15.117:6790/0 conn(0x7f58ec072400 legacy=0x7f58ec1e7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 shutdown_connections 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.370+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 >> 172.21.15.117:0/1642421406 conn(0x7f58ec067610 msgr2=0x7f58ec1436a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.371+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 shutdown_connections 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.371+0000 7f58f1a0d640 1 -- 172.21.15.117:0/1642421406 wait complete. 2024-03-29T11:22:33.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:33.371+0000 7f58f1a0d640 1 librados: shutdown 2024-03-29T11:22:33.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:40.169 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:40.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.336+0000 7f448e64c640 10 monclient: get_monmap_and_config 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.336+0000 7f448e64c640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.336+0000 7f448e64c640 10 monclient: monmap: 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:40.337583+0000 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:40.337583+0000 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.337+0000 7f448e64c640 1 Processor -- start 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.337+0000 7f448e64c640 1 -- start start 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.337+0000 7f448e64c640 10 monclient: init 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-c con 0x7f4480038c60 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-a con 0x7f44800324b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-b con 0x7f44800d11e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800caf80 con 0x7f44800324b0 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4480039850 con 0x7f4480038c60 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800c78d0 con 0x7f44800d11e0 2024-03-29T11:22:40.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.338+0000 7f448e64c640 10 monclient(hunting): authenticate will time out at 1028.647883s 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.339+0000 7f448d64a640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f4480038c60 0x7f44800c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:35428/0 (socket says 172.21.15.117:35428) 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.339+0000 7f448d64a640 1 -- 172.21.15.117:0/3337701599 learned_addr learned my addr 172.21.15.117:0/3337701599 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746613697 0 0) 0x7f4480039850 con 0x7f4480038c60 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:22:40.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.340+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4474003520 con 0x7f4480038c60 2024-03-29T11:22:40.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.341+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664630666 0 0) 0x7f4474003520 con 0x7f4480038c60 2024-03-29T11:22:40.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.341+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 >> v1:172.21.15.117:6790/0 conn(0x7f44800d11e0 legacy=0x7f44800d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:40.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 >> v1:172.21.15.117:6789/0 conn(0x7f44800324b0 legacy=0x7f44800cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 1 monclient: found mon.noname-c 2024-03-29T11:22:40.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.343+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44800c7a60 con 0x7f4480038c60 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f448e64c640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f448e64c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4478001690 con 0x7f4480038c60 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: dump: 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:40.322 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f448e64c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 1 -- 172.21.15.117:0/3337701599 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4478001920 con 0x7f4480038c60 2024-03-29T11:22:40.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f447ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f448e64c640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:40.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.344+0000 7f448e64c640 4 set_mon_vals no callback set 2024-03-29T11:22:40.324 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.345+0000 7f448e64c640 10 monclient: shutdown 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.345+0000 7f448e64c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.345+0000 7f448e64c640 1 -- 172.21.15.117:0/3337701599 >> v1:172.21.15.173:6789/0 conn(0x7f4480038c60 legacy=0x7f44800c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.345+0000 7f448e64c640 1 -- 172.21.15.117:0/3337701599 shutdown_connections 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.346+0000 7f448e64c640 1 -- 172.21.15.117:0/3337701599 >> 172.21.15.117:0/3337701599 conn(0x7f4480005c20 msgr2=0x7f4480006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.346+0000 7f448e64c640 1 -- 172.21.15.117:0/3337701599 shutdown_connections 2024-03-29T11:22:40.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.346+0000 7f448e64c640 1 -- 172.21.15.117:0/3337701599 wait complete. 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 10 monclient: build_initial_monmap 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 10 monclient: monmap: 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:40.326 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 librados: starting msgr at 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 librados: starting objecter 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 Processor -- start 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 -- start start 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 librados: setting wanted keys 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 1 librados: calling monclient init 2024-03-29T11:22:40.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.347+0000 7f448e64c640 10 monclient: init 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-c con 0x7f44800324b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-a con 0x7f4480038c60 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient(hunting): picked mon.noname-b con 0x7f44800d11e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800369c0 con 0x7f4480038c60 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4480037110 con 0x7f44800d11e0 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44800c2790 con 0x7f44800324b0 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:40.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.348+0000 7f448e64c640 10 monclient(hunting): authenticate will time out at 1028.657785s 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.351+0000 7f448ce49640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4480038c60 0x7f448015c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:39107/0 (socket says 172.21.15.117:39107) 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.351+0000 7f448ce49640 1 -- 172.21.15.117:0/1699978238 learned_addr learned my addr 172.21.15.117:0/1699978238 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 client.?.objecter ms_handle_connect 0x7f4480038c60 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 client.?.objecter ms_handle_connect 0x7f44800d11e0 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441078837 0 0) 0x7f4480037110 con 0x7f44800d11e0 2024-03-29T11:22:40.334 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c004eb0 con 0x7f44800d11e0 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.352+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367998538 0 0) 0x7f44800369c0 con 0x7f4480038c60 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4480037110 con 0x7f4480038c60 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4002491124 0 0) 0x7f446c004eb0 con 0x7f44800d11e0 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44800369c0 con 0x7f44800d11e0 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f448804fdd0 con 0x7f44800d11e0 2024-03-29T11:22:40.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): dump: 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3339245791 0 0) 0x7f4480037110 con 0x7f4480038c60 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.353+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f446c004eb0 con 0x7f4480038c60 2024-03-29T11:22:40.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4484001910 con 0x7f4480038c60 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient(hunting): dump: 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2976490029 0 0) 0x7f446c004eb0 con 0x7f4480038c60 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 >> v1:172.21.15.117:6790/0 conn(0x7f44800324b0 legacy=0x7f44800bc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 >> v1:172.21.15.173:6789/0 conn(0x7f44800d11e0 legacy=0x7f4480167e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 monclient: found mon.a 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4480036d70 con 0x7f4480038c60 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f447dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f448e64c640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f448e64c640 10 monclient: _renew_subs 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f448e64c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:22:40.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.354+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4480036370 con 0x7f4480038c60 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.355+0000 7f448e64c640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.355+0000 7f448e64c640 10 monclient: _renew_subs 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.355+0000 7f448e64c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.355+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44800c9b90 con 0x7f4480038c60 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.355+0000 7f448e64c640 1 librados: init done 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4484001a70 con 0x7f4480038c60 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447f7fe640 4 set_mon_vals no callback set 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f44840038b0 con 0x7f4480038c60 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.356+0000 7f447dffb640 10 monclient: dump: 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:40.339 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.357+0000 7f448e64c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.357+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4484003ad0 con 0x7f4480038c60 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.357+0000 7f448e64c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.357+0000 7f448e64c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.357+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468005080 con 0x7f4480038c60 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f4484085ac0 con 0x7f4480038c60 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 10 client.4353.objecter ms_dispatch 0x7f4488000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:22:40.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f448d64a640 10 monclient: get_auth_request con 0x7f446c06dd40 auth_method 0 2024-03-29T11:22:40.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.360+0000 7f447dffb640 10 client.4353.objecter ms_handle_connect 0x7f446c06dd40 2024-03-29T11:22:40.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.366+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4484050370 con 0x7f4480038c60 2024-03-29T11:22:40.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.366+0000 7f447dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:40.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.366+0000 7f447dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:40.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.692+0000 7f448e64c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.692+0000 7f448e64c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.692+0000 7f448e64c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:22:40.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.692+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4468005330 con 0x7f4480038c60 2024-03-29T11:22:40.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.693+0000 7f447dffb640 1 -- 172.21.15.117:0/1699978238 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f44840522a0 con 0x7f4480038c60 2024-03-29T11:22:40.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.693+0000 7f447dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:40.670 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.693+0000 7f447dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:40.671 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:40.671 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:40.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.697+0000 7f448e64c640 10 librados: watch_flush enter 2024-03-29T11:22:40.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.697+0000 7f448e64c640 10 librados: watch_flush exit 2024-03-29T11:22:40.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.697+0000 7f448e64c640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:22:40.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.697+0000 7f448e64c640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:22:40.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 >> v1:172.21.15.117:6816/4276038175 conn(0x7f446c06dd40 legacy=0x7f446c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 10 monclient: shutdown 2024-03-29T11:22:40.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:40.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 >> v1:172.21.15.117:6789/0 conn(0x7f4480038c60 legacy=0x7f448015c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:40.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 shutdown_connections 2024-03-29T11:22:40.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 >> 172.21.15.117:0/1699978238 conn(0x7f4480005c20 msgr2=0x7f44800043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:40.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 shutdown_connections 2024-03-29T11:22:40.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 -- 172.21.15.117:0/1699978238 wait complete. 2024-03-29T11:22:40.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:40.699+0000 7f448e64c640 1 librados: shutdown 2024-03-29T11:22:40.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:47.503 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:47.647 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.670+0000 7f3dd8de8640 10 monclient: get_monmap_and_config 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.670+0000 7f3dd8de8640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.670+0000 7f3dd8de8640 10 monclient: monmap: 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:47.671592+0000 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:47.671592+0000 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:47.648 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.649 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.672+0000 7f3dd8de8640 1 Processor -- start 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.673+0000 7f3dd8de8640 1 -- start start 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.673+0000 7f3dd8de8640 10 monclient: init 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-b con 0x7f3dcc006a70 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc02b100 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc029c30 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc02a270 con 0x7f3dcc029c30 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc02c440 con 0x7f3dcc02b100 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc0c9440 con 0x7f3dcc006a70 2024-03-29T11:22:47.653 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.674+0000 7f3dd8de8640 10 monclient(hunting): authenticate will time out at 1035.983664s 2024-03-29T11:22:47.654 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.677+0000 7f3dd37fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f3dcc006a70 0x7f3dcc0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58205/0 (socket says 172.21.15.117:58205) 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.677+0000 7f3dd37fe640 1 -- 172.21.15.117:0/2536470417 learned_addr learned my addr 172.21.15.117:0/2536470417 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674433309 0 0) 0x7f3dcc0c9440 con 0x7f3dcc006a70 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:22:47.655 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc4003520 con 0x7f3dcc006a70 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3067548972 0 0) 0x7f3dc4003520 con 0x7f3dcc006a70 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc02b100 legacy=0x7f3dcc0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 >> v1:172.21.15.117:6789/0 conn(0x7f3dcc029c30 legacy=0x7f3dcc0d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0c95d0 con 0x7f3dcc006a70 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3dc8001690 con 0x7f3dcc006a70 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.656 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: dump: 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd8de8640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd8de8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dc8001920 con 0x7f3dcc006a70 2024-03-29T11:22:47.657 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 1 -- 172.21.15.117:0/2536470417 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3dc8001cb0 con 0x7f3dcc006a70 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.678+0000 7f3dd27fc640 10 monclient: dump: 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:47.658 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.680+0000 7f3dd8de8640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:47.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.680+0000 7f3dd8de8640 4 set_mon_vals no callback set 2024-03-29T11:22:47.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.681+0000 7f3dd8de8640 10 monclient: shutdown 2024-03-29T11:22:47.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.681+0000 7f3dd8de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:47.659 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.681+0000 7f3dd8de8640 1 -- 172.21.15.117:0/2536470417 >> v1:172.21.15.117:6790/0 conn(0x7f3dcc006a70 legacy=0x7f3dcc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.686+0000 7f3dd8de8640 1 -- 172.21.15.117:0/2536470417 shutdown_connections 2024-03-29T11:22:47.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.686+0000 7f3dd8de8640 1 -- 172.21.15.117:0/2536470417 >> 172.21.15.117:0/2536470417 conn(0x7f3dcc0058f0 msgr2=0x7f3dcc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:47.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.686+0000 7f3dd8de8640 1 -- 172.21.15.117:0/2536470417 shutdown_connections 2024-03-29T11:22:47.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.686+0000 7f3dd8de8640 1 -- 172.21.15.117:0/2536470417 wait complete. 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.687+0000 7f3dd8de8640 10 monclient: build_initial_monmap 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.687+0000 7f3dd8de8640 10 monclient: monmap: 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:47.665 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.687+0000 7f3dd8de8640 1 librados: starting msgr at 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.687+0000 7f3dd8de8640 1 librados: starting objecter 2024-03-29T11:22:47.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.687+0000 7f3dd8de8640 1 Processor -- start 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.688+0000 7f3dd8de8640 1 -- start start 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.688+0000 7f3dd8de8640 1 librados: setting wanted keys 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.688+0000 7f3dd8de8640 1 librados: calling monclient init 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.688+0000 7f3dd8de8640 10 monclient: init 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.689+0000 7f3dd8de8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.689+0000 7f3dd8de8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:47.667 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.689+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc006a70 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc029c30 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 10 monclient(hunting): picked mon.noname-b con 0x7f3dcc1644d0 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc0cc410 con 0x7f3dcc006a70 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc163fd0 con 0x7f3dcc1644d0 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc0cd160 con 0x7f3dcc029c30 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:47.669 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.691+0000 7f3dd8de8640 10 monclient(hunting): authenticate will time out at 1036.000884s 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.692+0000 7f3dd37fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f3dcc006a70 0x7f3dcc0cb060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47627/0 (socket says 172.21.15.117:47627) 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.692+0000 7f3dd2ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f3dcc029c30 0x7f3dcc0ca300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58339/0 (socket says 172.21.15.117:58339) 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.692+0000 7f3dd2ffd640 1 -- 172.21.15.117:0/208264634 learned_addr learned my addr 172.21.15.117:0/208264634 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter ms_handle_connect 0x7f3dcc029c30 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter ms_handle_connect 0x7f3dcc006a70 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter ms_handle_connect 0x7f3dcc1644d0 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191538304 0 0) 0x7f3dcc0cd160 con 0x7f3dcc029c30 2024-03-29T11:22:47.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db8004eb0 con 0x7f3dcc029c30 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136285395 0 0) 0x7f3dcc163fd0 con 0x7f3dcc1644d0 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dcc0cd160 con 0x7f3dcc1644d0 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066203401 0 0) 0x7f3db8004eb0 con 0x7f3dcc029c30 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dcc163fd0 con 0x7f3dcc029c30 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3dc0001750 con 0x7f3dcc029c30 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:47.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): dump: 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.693+0000 7f3dbffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dd3fff640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc1644d0 legacy=0x7f3dcc031c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dd3fff640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc1644d0 legacy=0x7f3dcc031c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dd3fff640 1 --1- 172.21.15.117:0/208264634 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc1644d0 0x7f3dcc031c00 :-1 s=OPENED pgs=470077 cs=1 l=1).handle_message read tag failed 2024-03-29T11:22:47.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dd3fff640 1 --1- 172.21.15.117:0/208264634 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc1644d0 0x7f3dcc031c00 :-1 s=OPENED pgs=470077 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6789/0 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1346231481 0 0) 0x7f3dcc163fd0 con 0x7f3dcc029c30 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.173:6789/0 conn(0x7f3dcc1644d0 legacy=0x7f3dcc031c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.117:6789/0 conn(0x7f3dcc006a70 legacy=0x7f3dcc0cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 1 monclient: found mon.c 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0cd2f0 con 0x7f3dcc029c30 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.694+0000 7f3dbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dc0001ce0 con 0x7f3dcc029c30 2024-03-29T11:22:47.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3dc0004830 con 0x7f3dcc029c30 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dbffff640 10 monclient: dump: 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:47.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:22:47.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 10 monclient: _renew_subs 2024-03-29T11:22:47.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc164980 con 0x7f3dcc029c30 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 10 monclient: _renew_subs 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc164e10 con 0x7f3dcc029c30 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.695+0000 7f3dd8de8640 1 librados: init done 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.696+0000 7f3dd17fa640 4 set_mon_vals no callback set 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.697+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f3dc001a800 con 0x7f3dcc029c30 2024-03-29T11:22:47.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dd37fe640 10 monclient: get_auth_request con 0x7f3db806dc60 auth_method 0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dd8de8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dd8de8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dd8de8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db4005080 con 0x7f3dcc029c30 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f3dc0084820 con 0x7f3dcc029c30 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dbffff640 10 client.4391.objecter ms_dispatch 0x7f3dd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dbffff640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.698+0000 7f3dbffff640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.699+0000 7f3dbffff640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:22:47.683 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.699+0000 7f3dbffff640 10 client.4391.objecter ms_handle_connect 0x7f3db806dc60 2024-03-29T11:22:47.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.708+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3dc004ff30 con 0x7f3dcc029c30 2024-03-29T11:22:47.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.708+0000 7f3dbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:47.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:47.708+0000 7f3dbffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:48.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.035+0000 7f3dd8de8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.035+0000 7f3dd8de8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.035+0000 7f3dd8de8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:22:48.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.035+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3db4005330 con 0x7f3dcc029c30 2024-03-29T11:22:48.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.036+0000 7f3dbffff640 1 -- 172.21.15.117:0/208264634 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f3dc0052030 con 0x7f3dcc029c30 2024-03-29T11:22:48.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.036+0000 7f3dbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:48.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.036+0000 7f3dbffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:48.014 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:48.014 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:48.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.039+0000 7f3dd8de8640 10 librados: watch_flush enter 2024-03-29T11:22:48.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 10 librados: watch_flush exit 2024-03-29T11:22:48.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:22:48.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.117:6816/4276038175 conn(0x7f3db806dc60 legacy=0x7f3db8070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 10 monclient: shutdown 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 >> v1:172.21.15.117:6790/0 conn(0x7f3dcc029c30 legacy=0x7f3dcc0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 shutdown_connections 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.040+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 >> 172.21.15.117:0/208264634 conn(0x7f3dcc0058f0 msgr2=0x7f3dcc032d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.041+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 shutdown_connections 2024-03-29T11:22:48.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.041+0000 7f3dd8de8640 1 -- 172.21.15.117:0/208264634 wait complete. 2024-03-29T11:22:48.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:48.041+0000 7f3dd8de8640 1 librados: shutdown 2024-03-29T11:22:48.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:54.842 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:54.987 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.010+0000 7ff152f3b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.010+0000 7ff152f3b640 10 monclient: build_initial_monmap 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.010+0000 7ff152f3b640 10 monclient: monmap: 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:22:55.011247+0000 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:22:55.011247+0000 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:22:54.988 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:54.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.011+0000 7ff152f3b640 1 Processor -- start 2024-03-29T11:22:54.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.011+0000 7ff152f3b640 1 -- start start 2024-03-29T11:22:54.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.011+0000 7ff152f3b640 10 monclient: init 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c0a9750 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c13e1e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c071c30 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0ab300 con 0x7ff14c0a9750 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0ac2f0 con 0x7ff14c071c30 2024-03-29T11:22:54.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0ad8d0 con 0x7ff14c13e1e0 2024-03-29T11:22:54.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:54.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.012+0000 7ff152f3b640 10 monclient(hunting): authenticate will time out at 1043.321672s 2024-03-29T11:22:54.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.015+0000 7ff1514d8640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff14c071c30 0x7ff14c0ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:57320/0 (socket says 172.21.15.117:57320) 2024-03-29T11:22:54.992 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.015+0000 7ff1514d8640 1 -- 172.21.15.117:0/1136013002 learned_addr learned my addr 172.21.15.117:0/1136013002 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178824216 0 0) 0x7ff14c0ad8d0 con 0x7ff14c13e1e0 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:22:54.993 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff134003440 con 0x7ff14c13e1e0 2024-03-29T11:22:54.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688477089 0 0) 0x7ff134003440 con 0x7ff14c13e1e0 2024-03-29T11:22:54.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 >> v1:172.21.15.173:6789/0 conn(0x7ff14c071c30 legacy=0x7ff14c0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:54.994 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.016+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 >> v1:172.21.15.117:6789/0 conn(0x7ff14c0a9750 legacy=0x7ff14c13dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0ada60 con 0x7ff14c13e1e0 2024-03-29T11:22:54.996 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff140001690 con 0x7ff14c13e1e0 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff14b7fe640 10 monclient: dump: 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:54.997 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff152f3b640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.019+0000 7ff152f3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff140001860 con 0x7ff14c13e1e0 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 1 -- 172.21.15.117:0/1136013002 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff140001cb0 con 0x7ff14c13e1e0 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:54.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.020+0000 7ff14b7fe640 10 monclient: dump: 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.021+0000 7ff152f3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:54.999 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.021+0000 7ff152f3b640 4 set_mon_vals no callback set 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 10 monclient: shutdown 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 1 -- 172.21.15.117:0/1136013002 >> v1:172.21.15.117:6790/0 conn(0x7ff14c13e1e0 legacy=0x7ff14c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 1 -- 172.21.15.117:0/1136013002 shutdown_connections 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 1 -- 172.21.15.117:0/1136013002 >> 172.21.15.117:0/1136013002 conn(0x7ff14c06b660 msgr2=0x7ff14c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 1 -- 172.21.15.117:0/1136013002 shutdown_connections 2024-03-29T11:22:55.000 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.022+0000 7ff152f3b640 1 -- 172.21.15.117:0/1136013002 wait complete. 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.023+0000 7ff152f3b640 10 monclient: build_initial_monmap 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.023+0000 7ff152f3b640 10 monclient: monmap: 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 librados: starting msgr at 2024-03-29T11:22:55.004 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 librados: starting objecter 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 Processor -- start 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 -- start start 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 librados: setting wanted keys 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 1 librados: calling monclient init 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.024+0000 7ff152f3b640 10 monclient: init 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.025+0000 7ff152f3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.025+0000 7ff152f3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.025+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff14c071c30 addr v1:172.21.15.173:6789/0 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.025+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff14c0a9750 addr v1:172.21.15.117:6789/0 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff14c13e1e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c1414e0 con 0x7ff14c0a9750 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c13ff90 con 0x7ff14c071c30 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c0b2f00 con 0x7ff14c13e1e0 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:55.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.026+0000 7ff152f3b640 10 monclient(hunting): authenticate will time out at 1043.335866s 2024-03-29T11:22:55.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff150cd7640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff14c071c30 0x7ff14c135020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:57580/0 (socket says 172.21.15.117:57580) 2024-03-29T11:22:55.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff150cd7640 1 -- 172.21.15.117:0/3895258273 learned_addr learned my addr 172.21.15.117:0/3895258273 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:22:55.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 client.?.objecter ms_handle_connect 0x7ff14c071c30 2024-03-29T11:22:55.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979888515 0 0) 0x7ff14c13ff90 con 0x7ff14c071c30 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.028+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138004eb0 con 0x7ff14c071c30 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 324803200 0 0) 0x7ff138004eb0 con 0x7ff14c071c30 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff14c13ff90 con 0x7ff14c071c30 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff144001ce0 con 0x7ff14c071c30 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 10 monclient(hunting): dump: 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:55.007 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:55.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.029+0000 7ff1497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:55.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.031+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 793720761 0 0) 0x7ff14c13ff90 con 0x7ff14c071c30 2024-03-29T11:22:55.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.031+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 >> v1:172.21.15.117:6790/0 conn(0x7ff14c13e1e0 legacy=0x7ff14c1de7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 >> v1:172.21.15.117:6789/0 conn(0x7ff14c0a9750 legacy=0x7ff14c1d6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 1 monclient: found mon.b 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0b3090 con 0x7ff14c071c30 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 10 monclient: _renew_subs 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c138300 con 0x7ff14c071c30 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 10 monclient: _renew_subs 2024-03-29T11:22:55.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c135910 con 0x7ff14c071c30 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff152f3b640 1 librados: init done 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 client.4384.objecter ms_handle_connect 0x7ff14c13e1e0 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 client.4384.objecter resend_mon_ops 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff144001f40 con 0x7ff14c071c30 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff144003dc0 con 0x7ff14c071c30 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.032+0000 7ff1497fa640 10 monclient: dump: 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:55.013 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.033+0000 7ff14affd640 4 set_mon_vals no callback set 2024-03-29T11:22:55.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.033+0000 7ff152f3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.034+0000 7ff152f3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.034+0000 7ff152f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.034+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c005080 con 0x7ff14c071c30 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.035+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7ff144003f20 con 0x7ff14c071c30 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff14bfff640 10 monclient: get_auth_request con 0x7ff13806dc90 auth_method 0 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7ff144084f60 con 0x7ff14c071c30 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 10 client.4384.objecter ms_dispatch 0x7ff14c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:55.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:22:55.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.036+0000 7ff1497fa640 10 client.4384.objecter ms_handle_connect 0x7ff13806dc90 2024-03-29T11:22:55.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.044+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff144050670 con 0x7ff14c071c30 2024-03-29T11:22:55.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.044+0000 7ff1497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:55.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.044+0000 7ff1497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:55.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.381+0000 7ff152f3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.381+0000 7ff152f3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.381+0000 7ff152f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:22:55.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.381+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12c005330 con 0x7ff14c071c30 2024-03-29T11:22:55.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.383+0000 7ff1497fa640 1 -- 172.21.15.117:0/3895258273 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7ff144052770 con 0x7ff14c071c30 2024-03-29T11:22:55.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.383+0000 7ff1497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:55.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.383+0000 7ff1497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:55.361 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:22:55.362 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 10 librados: watch_flush enter 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 10 librados: watch_flush exit 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 >> v1:172.21.15.117:6816/4276038175 conn(0x7ff13806dc90 legacy=0x7ff138070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 10 monclient: shutdown 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 >> v1:172.21.15.173:6789/0 conn(0x7ff14c071c30 legacy=0x7ff14c135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 shutdown_connections 2024-03-29T11:22:55.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.387+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 >> 172.21.15.117:0/3895258273 conn(0x7ff14c06b660 msgr2=0x7ff14c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:55.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.390+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 shutdown_connections 2024-03-29T11:22:55.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.390+0000 7ff152f3b640 1 -- 172.21.15.117:0/3895258273 wait complete. 2024-03-29T11:22:55.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:22:55.390+0000 7ff152f3b640 1 librados: shutdown 2024-03-29T11:22:55.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:02.190 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:02.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.358+0000 7f46bd10e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.358+0000 7f46bd10e640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.358+0000 7f46bd10e640 10 monclient: monmap: 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:02.359192+0000 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:02.359192+0000 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:02.336 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 1 Processor -- start 2024-03-29T11:23:02.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 1 -- start start 2024-03-29T11:23:02.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient: init 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-c con 0x7f46b81422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-b con 0x7f46b8142690 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.359+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.360+0000 7f46bd10e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b8142c80 con 0x7f46b80b17d0 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.360+0000 7f46bd10e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80aae40 con 0x7f46b81422b0 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.360+0000 7f46bd10e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b813fbd0 con 0x7f46b8142690 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.360+0000 7f46bd10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.360+0000 7f46bd10e640 10 monclient(hunting): authenticate will time out at 1050.669124s 2024-03-29T11:23:02.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.362+0000 7f46b6d9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f46b81422b0 0x7f46b813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:36482/0 (socket says 172.21.15.117:36482) 2024-03-29T11:23:02.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.362+0000 7f46b6d9d640 1 -- 172.21.15.117:0/238087233 learned_addr learned my addr 172.21.15.117:0/238087233 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:02.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119530308 0 0) 0x7f46b813fbd0 con 0x7f46b8142690 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0003520 con 0x7f46b8142690 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.365+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383260257 0 0) 0x7f46a0003520 con 0x7f46b8142690 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 >> v1:172.21.15.173:6789/0 conn(0x7f46b81422b0 legacy=0x7f46b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 >> v1:172.21.15.117:6789/0 conn(0x7f46b80b17d0 legacy=0x7f46b80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b8151940 con 0x7f46b8142690 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f46a8001690 con 0x7f46b8142690 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.366+0000 7f46b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: dump: 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:02.346 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46bd10e640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46bd10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a80017f0 con 0x7f46b8142690 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 1 -- 172.21.15.117:0/238087233 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f46a8001c40 con 0x7f46b8142690 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46b5d9b640 10 monclient: dump: 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.347 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46bd10e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:02.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.367+0000 7f46bd10e640 4 set_mon_vals no callback set 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.369+0000 7f46bd10e640 10 monclient: shutdown 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.369+0000 7f46bd10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.369+0000 7f46bd10e640 1 -- 172.21.15.117:0/238087233 >> v1:172.21.15.117:6790/0 conn(0x7f46b8142690 legacy=0x7f46b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.370+0000 7f46bd10e640 1 -- 172.21.15.117:0/238087233 shutdown_connections 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.370+0000 7f46bd10e640 1 -- 172.21.15.117:0/238087233 >> 172.21.15.117:0/238087233 conn(0x7f46b8066a00 msgr2=0x7f46b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.370+0000 7f46bd10e640 1 -- 172.21.15.117:0/238087233 shutdown_connections 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.370+0000 7f46bd10e640 1 -- 172.21.15.117:0/238087233 wait complete. 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.371+0000 7f46bd10e640 10 monclient: build_initial_monmap 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.371+0000 7f46bd10e640 10 monclient: monmap: 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:02.350 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 librados: starting msgr at 2024-03-29T11:23:02.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 librados: starting objecter 2024-03-29T11:23:02.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 Processor -- start 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 -- start start 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 librados: setting wanted keys 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 1 librados: calling monclient init 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.372+0000 7f46bd10e640 10 monclient: init 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.373+0000 7f46bd10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-c con 0x7f46b80b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-a con 0x7f46b81422b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient(hunting): picked mon.noname-b con 0x7f46b8142690 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80ac730 con 0x7f46b81422b0 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80ac9a0 con 0x7f46b8142690 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80acc80 con 0x7f46b80b17d0 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:02.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.374+0000 7f46bd10e640 10 monclient(hunting): authenticate will time out at 1050.683718s 2024-03-29T11:23:02.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.376+0000 7f46b759e640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f46b8142690 0x7f46b81304d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:40608/0 (socket says 172.21.15.117:40608) 2024-03-29T11:23:02.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.376+0000 7f46b759e640 1 -- 172.21.15.117:0/4011192401 learned_addr learned my addr 172.21.15.117:0/4011192401 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:02.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.376+0000 7f469f7fe640 10 client.?.objecter ms_handle_connect 0x7f46b8142690 2024-03-29T11:23:02.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.376+0000 7f469f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770259160 0 0) 0x7f46b80ac9a0 con 0x7f46b8142690 2024-03-29T11:23:02.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:23:02.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.378+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a4004eb0 con 0x7f46b8142690 2024-03-29T11:23:02.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 client.?.objecter ms_handle_connect 0x7f46b80b17d0 2024-03-29T11:23:02.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916837478 0 0) 0x7f46b80acc80 con 0x7f46b80b17d0 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b80ac9a0 con 0x7f46b80b17d0 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621015098 0 0) 0x7f46a4004eb0 con 0x7f46b8142690 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b80acc80 con 0x7f46b8142690 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 client.?.objecter ms_handle_connect 0x7f46b81422b0 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f46b0004380 con 0x7f46b8142690 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.379+0000 7f469f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:02.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946486903 0 0) 0x7f46b80ac730 con 0x7f46b81422b0 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a4004eb0 con 0x7f46b81422b0 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342495937 0 0) 0x7f46b80ac9a0 con 0x7f46b80b17d0 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b80ac730 con 0x7f46b80b17d0 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f46ac0018a0 con 0x7f46b80b17d0 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.363 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 479330451 0 0) 0x7f46b80acc80 con 0x7f46b8142690 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 >> v1:172.21.15.117:6790/0 conn(0x7f46b80b17d0 legacy=0x7f46b8129cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 >> v1:172.21.15.117:6789/0 conn(0x7f46b81422b0 legacy=0x7f46b812fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 monclient: found mon.b 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b80abbf0 con 0x7f46b8142690 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:02.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f469f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 10 monclient: _renew_subs 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b8132660 con 0x7f46b8142690 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 10 monclient: _renew_subs 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b8132230 con 0x7f46b8142690 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.380+0000 7f46bd10e640 1 librados: init done 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.381+0000 7f46bd10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.381+0000 7f46bd10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.381+0000 7f46bd10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.381+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b80aba00 con 0x7f46b8142690 2024-03-29T11:23:02.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46b0001250 con 0x7f46b8142690 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f46b0001620 con 0x7f46b8142690 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f469f7fe640 10 monclient: dump: 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:02.366 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.382+0000 7f46b559a640 4 set_mon_vals no callback set 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.386+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f46b0001780 con 0x7f46b8142690 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f46b0084ca0 con 0x7f46b8142690 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f469f7fe640 10 client.4390.objecter ms_dispatch 0x7f46b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:02.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f469f7fe640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:02.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f469f7fe640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:02.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f46b6d9d640 10 monclient: get_auth_request con 0x7f46a406df70 auth_method 0 2024-03-29T11:23:02.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.387+0000 7f469f7fe640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:23:02.368 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.388+0000 7f469f7fe640 10 client.4390.objecter ms_handle_connect 0x7f46a406df70 2024-03-29T11:23:02.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.391+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46b0050580 con 0x7f46b8142690 2024-03-29T11:23:02.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.391+0000 7f469f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:02.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.391+0000 7f469f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:02.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.709+0000 7f46bd10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.709+0000 7f46bd10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.709+0000 7f46bd10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:02.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.709+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46b80abd80 con 0x7f46b8142690 2024-03-29T11:23:02.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.710+0000 7f469f7fe640 1 -- 172.21.15.117:0/4011192401 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f46b00524b0 con 0x7f46b8142690 2024-03-29T11:23:02.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.710+0000 7f469f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:02.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.710+0000 7f469f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:02.689 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:02.689 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.715+0000 7f46bd10e640 10 librados: watch_flush enter 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 10 librados: watch_flush exit 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 >> v1:172.21.15.117:6816/4276038175 conn(0x7f46a406df70 legacy=0x7f46a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 10 monclient: shutdown 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:02.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 >> v1:172.21.15.173:6789/0 conn(0x7f46b8142690 legacy=0x7f46b81304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:02.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 shutdown_connections 2024-03-29T11:23:02.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.716+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 >> 172.21.15.117:0/4011192401 conn(0x7f46b8066a00 msgr2=0x7f46b80b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:02.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.717+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 shutdown_connections 2024-03-29T11:23:02.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.717+0000 7f46bd10e640 1 -- 172.21.15.117:0/4011192401 wait complete. 2024-03-29T11:23:02.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:02.717+0000 7f46bd10e640 1 librados: shutdown 2024-03-29T11:23:02.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:09.518 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.709+0000 7fe5e616d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.709+0000 7fe5e616d640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.710+0000 7fe5e616d640 10 monclient: monmap: 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:09.711082+0000 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:09.711082+0000 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:09.688 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:09.689 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:09.689 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.713+0000 7fe5e616d640 1 Processor -- start 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.713+0000 7fe5e616d640 1 -- start start 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.713+0000 7fe5e616d640 10 monclient: init 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e00b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e0071eb0 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e0143d10 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e006ef80 con 0x7fe5e00b1f60 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e006f260 con 0x7fe5e0071eb0 2024-03-29T11:23:09.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00b40c0 con 0x7fe5e0143d10 2024-03-29T11:23:09.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.714+0000 7fe5e616d640 10 monclient(hunting): authenticate will time out at 1058.023896s 2024-03-29T11:23:09.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.721+0000 7fe5df7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe5e00b1f60 0x7fe5e0143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:47311/0 (socket says 172.21.15.117:47311) 2024-03-29T11:23:09.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.721+0000 7fe5df7fe640 1 -- 172.21.15.117:0/2139759072 learned_addr learned my addr 172.21.15.117:0/2139759072 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388356225 0 0) 0x7fe5e00b40c0 con 0x7fe5e0143d10 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c8003520 con 0x7fe5e0143d10 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956496876 0 0) 0x7fe5e006ef80 con 0x7fe5e00b1f60 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:23:09.704 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.722+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e00b40c0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258569248 0 0) 0x7fe5c8003520 con 0x7fe5e0143d10 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 >> v1:172.21.15.173:6789/0 conn(0x7fe5e0071eb0 legacy=0x7fe5e0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 >> v1:172.21.15.117:6789/0 conn(0x7fe5e00b1f60 legacy=0x7fe5e0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0155d70 con 0x7fe5e0143d10 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d4001690 con 0x7fe5e0143d10 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: dump: 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5d4001870 con 0x7fe5e0143d10 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 1 -- 172.21.15.117:0/2139759072 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d4001690 con 0x7fe5e0143d10 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.726+0000 7fe5de7fc640 10 monclient: dump: 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.727+0000 7fe5e616d640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:23:09.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.727+0000 7fe5e616d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.727+0000 7fe5e616d640 4 set_mon_vals no callback set 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.727+0000 7fe5e616d640 10 monclient: shutdown 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.728+0000 7fe5e616d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.728+0000 7fe5e616d640 1 -- 172.21.15.117:0/2139759072 >> v1:172.21.15.117:6790/0 conn(0x7fe5e0143d10 legacy=0x7fe5e0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.728+0000 7fe5e616d640 1 -- 172.21.15.117:0/2139759072 shutdown_connections 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.728+0000 7fe5e616d640 1 -- 172.21.15.117:0/2139759072 >> 172.21.15.117:0/2139759072 conn(0x7fe5e0066a00 msgr2=0x7fe5e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.732+0000 7fe5e616d640 1 -- 172.21.15.117:0/2139759072 shutdown_connections 2024-03-29T11:23:09.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.732+0000 7fe5e616d640 1 -- 172.21.15.117:0/2139759072 wait complete. 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.733+0000 7fe5e616d640 10 monclient: build_initial_monmap 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.733+0000 7fe5e616d640 10 monclient: monmap: 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:09.716 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 librados: starting msgr at 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 librados: starting objecter 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 Processor -- start 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 -- start start 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 librados: setting wanted keys 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 1 librados: calling monclient init 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.734+0000 7fe5e616d640 10 monclient: init 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.735+0000 7fe5e616d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:09.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.735+0000 7fe5e616d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.735+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e00b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.738+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e0143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.738+0000 7fe5e616d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e00b17f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.738+0000 7fe5e616d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00ad5f0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.738+0000 7fe5df7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe5e00b1f60 0x7fe5e00b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:38755/0 (socket says 172.21.15.117:38755) 2024-03-29T11:23:09.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.738+0000 7fe5df7fe640 1 -- 172.21.15.117:0/1973033211 learned_addr learned my addr 172.21.15.117:0/1973033211 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:09.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00ad8d0 con 0x7fe5e0143d10 2024-03-29T11:23:09.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00ac470 con 0x7fe5e00b17f0 2024-03-29T11:23:09.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5e616d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:09.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5e616d640 10 monclient(hunting): authenticate will time out at 1058.051457s 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5c7fff640 10 client.?.objecter ms_handle_connect 0x7fe5e00b1f60 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.742+0000 7fe5c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 client.?.objecter ms_handle_connect 0x7fe5e0143d10 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 client.?.objecter ms_handle_connect 0x7fe5e00b17f0 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969544447 0 0) 0x7fe5e00ad5f0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.743+0000 7fe5c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5cc004eb0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179780424 0 0) 0x7fe5e00ad8d0 con 0x7fe5e0143d10 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e00ad5f0 con 0x7fe5e0143d10 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436345232 0 0) 0x7fe5e00ac470 con 0x7fe5e00b17f0 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e00ad8d0 con 0x7fe5e00b17f0 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 473254288 0 0) 0x7fe5e00ad8d0 con 0x7fe5e00b17f0 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e00ac470 con 0x7fe5e00b17f0 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d4001690 con 0x7fe5e00b17f0 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:09.725 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978225874 0 0) 0x7fe5cc004eb0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e00ad8d0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d00018c0 con 0x7fe5e00b1f60 2024-03-29T11:23:09.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.744+0000 7fe5c7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 736969803 0 0) 0x7fe5e00ac470 con 0x7fe5e00b17f0 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 >> v1:172.21.15.173:6789/0 conn(0x7fe5e0143d10 legacy=0x7fe5e01e3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.727 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 >> v1:172.21.15.117:6789/0 conn(0x7fe5e00b1f60 legacy=0x7fe5e00b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 1 monclient: found mon.c 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00abb70 con 0x7fe5e00b17f0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 10 monclient: _renew_subs 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00ab270 con 0x7fe5e00b17f0 2024-03-29T11:23:09.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:09.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 10 monclient: _renew_subs 2024-03-29T11:23:09.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:09.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e01e4770 con 0x7fe5e00b17f0 2024-03-29T11:23:09.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.745+0000 7fe5e616d640 1 librados: init done 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.746+0000 7fe5e616d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.746+0000 7fe5e616d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.746+0000 7fe5e616d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.746+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e00abd60 con 0x7fe5e00b17f0 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5d4001d40 con 0x7fe5e00b17f0 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d4001240 con 0x7fe5e00b17f0 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:09.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5c7fff640 10 monclient: dump: 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.751+0000 7fe5ddffb640 4 set_mon_vals no callback set 2024-03-29T11:23:09.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.752+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fe5d40197d0 con 0x7fe5e00b17f0 2024-03-29T11:23:09.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5df7fe640 10 monclient: get_auth_request con 0x7fe5cc06e080 auth_method 0 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fe5d4083670 con 0x7fe5e00b17f0 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5c7fff640 10 client.4406.objecter ms_dispatch 0x7fe5e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5c7fff640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5c7fff640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.753+0000 7fe5c7fff640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:23:09.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.756+0000 7fe5c7fff640 10 client.4406.objecter ms_handle_connect 0x7fe5cc06e080 2024-03-29T11:23:09.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.760+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5d404ef50 con 0x7fe5e00b17f0 2024-03-29T11:23:09.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.760+0000 7fe5c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:09.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:09.760+0000 7fe5c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:10.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.085+0000 7fe5e616d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.085+0000 7fe5e616d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.085+0000 7fe5e616d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:10.064 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.085+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e0075f10 con 0x7fe5e00b17f0 2024-03-29T11:23:10.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.087+0000 7fe5c7fff640 1 -- 172.21.15.117:0/1973033211 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fe5d4050e80 con 0x7fe5e00b17f0 2024-03-29T11:23:10.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.087+0000 7fe5c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:10.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.087+0000 7fe5c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:10.067 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:10.067 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.090+0000 7fe5e616d640 10 librados: watch_flush enter 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 10 librados: watch_flush exit 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 >> v1:172.21.15.117:6816/4276038175 conn(0x7fe5cc06e080 legacy=0x7fe5cc070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 10 monclient: shutdown 2024-03-29T11:23:10.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:10.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.091+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 >> v1:172.21.15.117:6790/0 conn(0x7fe5e00b17f0 legacy=0x7fe5e01ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:10.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.095+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 shutdown_connections 2024-03-29T11:23:10.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.095+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 >> 172.21.15.117:0/1973033211 conn(0x7fe5e0066a00 msgr2=0x7fe5e0142860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:10.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.095+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 shutdown_connections 2024-03-29T11:23:10.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.095+0000 7fe5e616d640 1 -- 172.21.15.117:0/1973033211 wait complete. 2024-03-29T11:23:10.073 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:10.095+0000 7fe5e616d640 1 librados: shutdown 2024-03-29T11:23:10.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:16.897 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.072+0000 7f997050f640 10 monclient: get_monmap_and_config 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.072+0000 7f997050f640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.072+0000 7f997050f640 10 monclient: monmap: 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:17.073569+0000 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:17.073569+0000 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.051 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:17.052 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:17.052 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:17.052 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.073+0000 7f997050f640 1 Processor -- start 2024-03-29T11:23:17.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.073+0000 7f997050f640 1 -- start start 2024-03-29T11:23:17.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.073+0000 7f997050f640 10 monclient: init 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient(hunting): picked mon.noname-b con 0x7f99680b4710 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient(hunting): picked mon.noname-c con 0x7f99680b3970 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient(hunting): picked mon.noname-a con 0x7f996813f390 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680af020 con 0x7f996813f390 2024-03-29T11:23:17.060 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680b0910 con 0x7f99680b3970 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680b0bf0 con 0x7f99680b4710 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.074+0000 7f997050f640 10 monclient(hunting): authenticate will time out at 1065.383829s 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996e2ab640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f99680b4710 0x7f99680b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:32977/0 (socket says 172.21.15.117:32977) 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996e2ab640 1 -- 172.21.15.117:0/800129597 learned_addr learned my addr 172.21.15.117:0/800129597 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666059831 0 0) 0x7f99680b0bf0 con 0x7f99680b4710 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.078+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995400bd30 con 0x7f99680b4710 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.079+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2249041453 0 0) 0x7f995400bd30 con 0x7f99680b4710 2024-03-29T11:23:17.061 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.079+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 >> v1:172.21.15.173:6789/0 conn(0x7f99680b3970 legacy=0x7f99680af4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:17.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 >> v1:172.21.15.117:6789/0 conn(0x7f996813f390 legacy=0x7f996813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:17.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.065 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 1 monclient: found mon.noname-b 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680aeb00 con 0x7f99680b4710 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9958001690 con 0x7f99680b4710 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: dump: 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.066 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f997050f640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f997050f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99580018f0 con 0x7f99680b4710 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f996d2a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f997050f640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.082+0000 7f997050f640 4 set_mon_vals no callback set 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f996d2a9640 1 -- 172.21.15.117:0/800129597 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9958001d40 con 0x7f99680b4710 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f996d2a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f996d2a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:17.067 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f996d2a9640 10 monclient: dump: 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 10 monclient: shutdown 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 1 -- 172.21.15.117:0/800129597 >> v1:172.21.15.117:6790/0 conn(0x7f99680b4710 legacy=0x7f99680b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 1 -- 172.21.15.117:0/800129597 shutdown_connections 2024-03-29T11:23:17.068 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 1 -- 172.21.15.117:0/800129597 >> 172.21.15.117:0/800129597 conn(0x7f9968066a00 msgr2=0x7f9968066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 1 -- 172.21.15.117:0/800129597 shutdown_connections 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.083+0000 7f997050f640 1 -- 172.21.15.117:0/800129597 wait complete. 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.084+0000 7f997050f640 10 monclient: build_initial_monmap 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.084+0000 7f997050f640 10 monclient: monmap: 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.069 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.084+0000 7f997050f640 1 librados: starting msgr at 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.084+0000 7f997050f640 1 librados: starting objecter 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.085+0000 7f997050f640 1 Processor -- start 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.085+0000 7f997050f640 1 -- start start 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.085+0000 7f997050f640 1 librados: setting wanted keys 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.085+0000 7f997050f640 1 librados: calling monclient init 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.085+0000 7f997050f640 10 monclient: init 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.086+0000 7f997050f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.086+0000 7f997050f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:17.070 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.086+0000 7f997050f640 10 monclient(hunting): picked mon.noname-c con 0x7f99680b3970 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:17.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 10 monclient(hunting): picked mon.noname-a con 0x7f99680b4710 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:17.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 10 monclient(hunting): picked mon.noname-b con 0x7f996813f390 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:17.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99681424f0 con 0x7f99680b4710 2024-03-29T11:23:17.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9968141590 con 0x7f996813f390 2024-03-29T11:23:17.071 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9968142950 con 0x7f99680b3970 2024-03-29T11:23:17.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:17.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.088+0000 7f997050f640 10 monclient(hunting): authenticate will time out at 1065.397282s 2024-03-29T11:23:17.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.091+0000 7f996e2ab640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f99680b3970 0x7f9968138c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:33069/0 (socket says 172.21.15.117:33069) 2024-03-29T11:23:17.072 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.091+0000 7f996e2ab640 1 -- 172.21.15.117:0/366139831 learned_addr learned my addr 172.21.15.117:0/366139831 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 client.?.objecter ms_handle_connect 0x7f99680b3970 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 client.?.objecter ms_handle_connect 0x7f99680b4710 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379962994 0 0) 0x7f99681424f0 con 0x7f99680b4710 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:17.079 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.097+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c004eb0 con 0x7f99680b4710 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385377726 0 0) 0x7f994c004eb0 con 0x7f99680b4710 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99681424f0 con 0x7f99680b4710 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9960001bf0 con 0x7f99680b4710 2024-03-29T11:23:17.080 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient(hunting): dump: 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2127543932 0 0) 0x7f99681424f0 con 0x7f99680b4710 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 >> v1:172.21.15.117:6790/0 conn(0x7f99680b3970 legacy=0x7f9968138c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 >> v1:172.21.15.173:6789/0 conn(0x7f996813f390 legacy=0x7f99681d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 monclient: found mon.a 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:17.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968139380 con 0x7f99680b4710 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9960001d50 con 0x7f99680b4710 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9960003c30 con 0x7f99680b4710 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f995effd640 10 monclient: dump: 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:17.085 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.098+0000 7f996caa8640 4 set_mon_vals no callback set 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.100+0000 7f997050f640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.100+0000 7f997050f640 10 monclient: _renew_subs 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.100+0000 7f997050f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.100+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99681395d0 con 0x7f99680b4710 2024-03-29T11:23:17.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.101+0000 7f997050f640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.101+0000 7f997050f640 10 monclient: _renew_subs 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.101+0000 7f997050f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.101+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9968070850 con 0x7f99680b4710 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.101+0000 7f997050f640 1 librados: init done 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f9960019c00 con 0x7f99680b4710 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f9960083c20 con 0x7f99680b4710 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 10 client.4368.objecter ms_dispatch 0x7f9968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.103+0000 7f995effd640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.104+0000 7f996e2ab640 10 monclient: get_auth_request con 0x7f994c07ed10 auth_method 0 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.105+0000 7f995effd640 10 client.4368.objecter ms_handle_connect 0x7f994c07ed10 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.105+0000 7f997050f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.105+0000 7f997050f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.105+0000 7f997050f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:17.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.105+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99680688a0 con 0x7f99680b4710 2024-03-29T11:23:17.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.120+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f996004f330 con 0x7f99680b4710 2024-03-29T11:23:17.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.120+0000 7f995effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:17.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.120+0000 7f995effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:17.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.445+0000 7f997050f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.445+0000 7f997050f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.445+0000 7f997050f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:17.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.445+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9968075f10 con 0x7f99680b4710 2024-03-29T11:23:17.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.446+0000 7f995effd640 1 -- 172.21.15.117:0/366139831 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f9960051430 con 0x7f99680b4710 2024-03-29T11:23:17.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.446+0000 7f995effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:17.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.446+0000 7f995effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:17.426 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:17.430 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:17.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.452+0000 7f997050f640 10 librados: watch_flush enter 2024-03-29T11:23:17.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.452+0000 7f997050f640 10 librados: watch_flush exit 2024-03-29T11:23:17.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.452+0000 7f997050f640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:23:17.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.452+0000 7f997050f640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:23:17.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.452+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 >> v1:172.21.15.117:6816/4276038175 conn(0x7f994c07ed10 legacy=0x7f994c0811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 10 monclient: shutdown 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 >> v1:172.21.15.117:6789/0 conn(0x7f99680b4710 legacy=0x7f99681d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 shutdown_connections 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 >> 172.21.15.117:0/366139831 conn(0x7f9968066a00 msgr2=0x7f99680b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 shutdown_connections 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 -- 172.21.15.117:0/366139831 wait complete. 2024-03-29T11:23:17.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:17.453+0000 7f997050f640 1 librados: shutdown 2024-03-29T11:23:17.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:24.256 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:24.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.414+0000 7f45c140f640 10 monclient: get_monmap_and_config 2024-03-29T11:23:24.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.414+0000 7f45c140f640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.414+0000 7f45c140f640 10 monclient: monmap: 2024-03-29T11:23:24.393 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:24.393 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:24.415625+0000 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:24.415625+0000 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.416+0000 7f45c140f640 1 Processor -- start 2024-03-29T11:23:24.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.416+0000 7f45c140f640 1 -- start start 2024-03-29T11:23:24.395 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.416+0000 7f45c140f640 10 monclient: init 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-c con 0x7f45bc070170 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-a con 0x7f45bc0713f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-b con 0x7f45bc070550 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc13dbd0 con 0x7f45bc0713f0 2024-03-29T11:23:24.397 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc13ff70 con 0x7f45bc070170 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc140250 con 0x7f45bc070550 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45c140f640 10 monclient(hunting): authenticate will time out at 1072.726102s 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45bb7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f45bc070550 0x7f45bc14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:56612/0 (socket says 172.21.15.117:56612) 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.417+0000 7f45bb7fe640 1 -- 172.21.15.117:0/3806655842 learned_addr learned my addr 172.21.15.117:0/3806655842 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456524342 0 0) 0x7f45bc140250 con 0x7f45bc070550 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a8003520 con 0x7f45bc070550 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023304192 0 0) 0x7f45a8003520 con 0x7f45bc070550 2024-03-29T11:23:24.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.418+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 >> v1:172.21.15.173:6789/0 conn(0x7f45bc070170 legacy=0x7f45bc13d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 >> v1:172.21.15.117:6789/0 conn(0x7f45bc0713f0 legacy=0x7f45bc1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc140df0 con 0x7f45bc070550 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f45b4001a10 con 0x7f45bc070550 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45ba7fc640 10 monclient: dump: 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45c140f640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:23:24.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.420+0000 7f45c140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45b4001be0 con 0x7f45bc070550 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 1 -- 172.21.15.117:0/3806655842 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f45b4001ff0 con 0x7f45bc070550 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45ba7fc640 10 monclient: dump: 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.401 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45c140f640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.421+0000 7f45c140f640 4 set_mon_vals no callback set 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.422+0000 7f45c140f640 10 monclient: shutdown 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.422+0000 7f45c140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.422+0000 7f45c140f640 1 -- 172.21.15.117:0/3806655842 >> v1:172.21.15.117:6790/0 conn(0x7f45bc070550 legacy=0x7f45bc14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.422+0000 7f45c140f640 1 -- 172.21.15.117:0/3806655842 shutdown_connections 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.422+0000 7f45c140f640 1 -- 172.21.15.117:0/3806655842 >> 172.21.15.117:0/3806655842 conn(0x7f45bc067610 msgr2=0x7f45bc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.423+0000 7f45c140f640 1 -- 172.21.15.117:0/3806655842 shutdown_connections 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.423+0000 7f45c140f640 1 -- 172.21.15.117:0/3806655842 wait complete. 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.424+0000 7f45c140f640 10 monclient: build_initial_monmap 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.424+0000 7f45c140f640 10 monclient: monmap: 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:24.403 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.424+0000 7f45c140f640 1 librados: starting msgr at 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.424+0000 7f45c140f640 1 librados: starting objecter 2024-03-29T11:23:24.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.425+0000 7f45c140f640 1 Processor -- start 2024-03-29T11:23:24.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.426+0000 7f45c140f640 1 -- start start 2024-03-29T11:23:24.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.426+0000 7f45c140f640 1 librados: setting wanted keys 2024-03-29T11:23:24.406 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.426+0000 7f45c140f640 1 librados: calling monclient init 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.426+0000 7f45c140f640 10 monclient: init 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-a con 0x7f45bc070170 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-c con 0x7f45bc070550 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient(hunting): picked mon.noname-b con 0x7f45bc0713f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc1d3690 con 0x7f45bc070170 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc13dd70 con 0x7f45bc0713f0 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc13df00 con 0x7f45bc070550 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:24.407 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.427+0000 7f45c140f640 10 monclient(hunting): authenticate will time out at 1072.736799s 2024-03-29T11:23:24.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.428+0000 7f45b3fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f45bc070550 0x7f45bc1d3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:49388/0 (socket says 172.21.15.117:49388) 2024-03-29T11:23:24.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.428+0000 7f45b3fff640 1 -- 172.21.15.117:0/1006750763 learned_addr learned my addr 172.21.15.117:0/1006750763 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:24.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.428+0000 7f45b37fe640 10 client.?.objecter ms_handle_connect 0x7f45bc070170 2024-03-29T11:23:24.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.428+0000 7f45b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.429+0000 7f45b37fe640 10 client.?.objecter ms_handle_connect 0x7f45bc070550 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.429+0000 7f45b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.429+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642924722 0 0) 0x7f45bc1d3690 con 0x7f45bc070170 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:24.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c004eb0 con 0x7f45bc070170 2024-03-29T11:23:24.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.430+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595612777 0 0) 0x7f45ac0011b0 con 0x7f45bc070550 2024-03-29T11:23:24.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45bc1d3690 con 0x7f45bc070550 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1630332556 0 0) 0x7f459c004eb0 con 0x7f45bc070170 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45ac0011b0 con 0x7f45bc070170 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f45a40017d0 con 0x7f45bc070170 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): dump: 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:24.414 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.431+0000 7f45b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 10 client.?.objecter ms_handle_connect 0x7f45bc0713f0 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:24.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612801275 0 0) 0x7f45bc1d3690 con 0x7f45bc070550 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f459c004eb0 con 0x7f45bc070550 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f45ac001920 con 0x7f45bc070550 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 10 monclient(hunting): dump: 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2681046864 0 0) 0x7f45ac0011b0 con 0x7f45bc070170 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 >> v1:172.21.15.117:6790/0 conn(0x7f45bc070550 legacy=0x7f45bc1d3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.432+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 >> v1:172.21.15.173:6789/0 conn(0x7f45bc0713f0 legacy=0x7f45bc132c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 1 monclient: found mon.a 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc133960 con 0x7f45bc070170 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 10 monclient: _renew_subs 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc139210 con 0x7f45bc070170 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:24.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 10 monclient: _renew_subs 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc137f60 con 0x7f45bc070170 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.433+0000 7f45c140f640 1 librados: init done 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45a4001ce0 con 0x7f45bc070170 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f45a4003790 con 0x7f45bc070170 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.434+0000 7f45b37fe640 10 monclient: dump: 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:24.418 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:24.419 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:24.419 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:24.419 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.435+0000 7f45b9ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.436+0000 7f45c140f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.436+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f45a4019760 con 0x7f45bc070170 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.436+0000 7f45c140f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.436+0000 7f45c140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.436+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a0005080 con 0x7f45bc070170 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.437+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f45a4083710 con 0x7f45bc070170 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.437+0000 7f45b37fe640 10 client.4374.objecter ms_dispatch 0x7f45bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.437+0000 7f45b37fe640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.437+0000 7f45b37fe640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.437+0000 7f45b37fe640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T11:23:24.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.438+0000 7f45b3fff640 10 monclient: get_auth_request con 0x7f459c06de40 auth_method 0 2024-03-29T11:23:24.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.438+0000 7f45b37fe640 10 client.4374.objecter ms_handle_connect 0x7f459c06de40 2024-03-29T11:23:24.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.445+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45a404eff0 con 0x7f45bc070170 2024-03-29T11:23:24.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.445+0000 7f45b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:24.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.445+0000 7f45b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:24.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.769+0000 7f45c140f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.769+0000 7f45c140f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.769+0000 7f45c140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:24.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.769+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45a0005330 con 0x7f45bc070170 2024-03-29T11:23:24.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.773+0000 7f45b37fe640 1 -- 172.21.15.117:0/1006750763 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f45a4050f20 con 0x7f45bc070170 2024-03-29T11:23:24.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.773+0000 7f45b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:24.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.773+0000 7f45b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:24.752 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:24.752 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.776+0000 7f45c140f640 10 librados: watch_flush enter 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.776+0000 7f45c140f640 10 librados: watch_flush exit 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.776+0000 7f45c140f640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.776+0000 7f45c140f640 10 client.4374.objecter successfully canceled tick 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.777+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 >> v1:172.21.15.117:6816/4276038175 conn(0x7f459c06de40 legacy=0x7f459c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.777+0000 7f45c140f640 10 monclient: shutdown 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.777+0000 7f45c140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:24.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.777+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 >> v1:172.21.15.117:6789/0 conn(0x7f45bc070170 legacy=0x7f45bc138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:24.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.779+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 shutdown_connections 2024-03-29T11:23:24.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.779+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 >> 172.21.15.117:0/1006750763 conn(0x7f45bc067610 msgr2=0x7f45bc0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:24.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.779+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 shutdown_connections 2024-03-29T11:23:24.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.779+0000 7f45c140f640 1 -- 172.21.15.117:0/1006750763 wait complete. 2024-03-29T11:23:24.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:24.779+0000 7f45c140f640 1 librados: shutdown 2024-03-29T11:23:24.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:31.583 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.768+0000 7f335a0a0640 10 monclient: get_monmap_and_config 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.768+0000 7f335a0a0640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.768+0000 7f335a0a0640 10 monclient: monmap: 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:31.769712+0000 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:31.769712+0000 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:31.748 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.770+0000 7f335a0a0640 1 Processor -- start 2024-03-29T11:23:31.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.770+0000 7f335a0a0640 1 -- start start 2024-03-29T11:23:31.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.770+0000 7f335a0a0640 10 monclient: init 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-c con 0x7f334c0389e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-a con 0x7f334c038dc0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335909e640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f334c0389e0 0x7f334c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:59424/0 (socket says 172.21.15.117:59424) 2024-03-29T11:23:31.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-b con 0x7f334c030660 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335909e640 1 -- 172.21.15.117:0/1266979052 learned_addr learned my addr 172.21.15.117:0/1266979052 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0c54c0 con 0x7f334c038dc0 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c036430 con 0x7f334c0389e0 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.771+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c039840 con 0x7f334c030660 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f335a0a0640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f335a0a0640 10 monclient(hunting): authenticate will time out at 1080.081056s 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107483174 0 0) 0x7f334c036430 con 0x7f334c0389e0 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:23:31.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3340003520 con 0x7f334c0389e0 2024-03-29T11:23:31.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2019054251 0 0) 0x7f3340003520 con 0x7f334c0389e0 2024-03-29T11:23:31.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 >> v1:172.21.15.117:6790/0 conn(0x7f334c030660 legacy=0x7f334c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 >> v1:172.21.15.117:6789/0 conn(0x7f334c038dc0 legacy=0x7f334c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:31.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 monclient: found mon.noname-c 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c035690 con 0x7f334c0389e0 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3350001690 con 0x7f334c0389e0 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.772+0000 7f334bfff640 10 monclient: dump: 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.754 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f335a0a0640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f335a0a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3350001860 con 0x7f334c0389e0 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 1 -- 172.21.15.117:0/1266979052 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3350001c70 con 0x7f334c0389e0 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f334bfff640 10 monclient: dump: 2024-03-29T11:23:31.755 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f335a0a0640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:31.756 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.773+0000 7f335a0a0640 4 set_mon_vals no callback set 2024-03-29T11:23:31.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.774+0000 7f335a0a0640 10 monclient: shutdown 2024-03-29T11:23:31.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.774+0000 7f335a0a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:31.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.774+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 >> v1:172.21.15.173:6789/0 conn(0x7f334c0389e0 legacy=0x7f334c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.775+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 shutdown_connections 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.775+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 >> 172.21.15.117:0/1266979052 conn(0x7f334c005c00 msgr2=0x7f334c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.775+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 shutdown_connections 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.775+0000 7f335a0a0640 1 -- 172.21.15.117:0/1266979052 wait complete. 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.776+0000 7f335a0a0640 10 monclient: build_initial_monmap 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.776+0000 7f335a0a0640 10 monclient: monmap: 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:31.759 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.776+0000 7f335a0a0640 1 librados: starting msgr at 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.776+0000 7f335a0a0640 1 librados: starting objecter 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.777+0000 7f335a0a0640 1 Processor -- start 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.777+0000 7f335a0a0640 1 -- start start 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.777+0000 7f335a0a0640 1 librados: setting wanted keys 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.777+0000 7f335a0a0640 1 librados: calling monclient init 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.777+0000 7f335a0a0640 10 monclient: init 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-a con 0x7f334c030660 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-b con 0x7f334c0389e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 10 monclient(hunting): picked mon.noname-c con 0x7f334c02d6e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c02eba0 con 0x7f334c030660 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.778+0000 7f335a0a0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c02dd10 con 0x7f334c0389e0 2024-03-29T11:23:31.760 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f335889d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f334c0389e0 0x7f334c0c0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:59942/0 (socket says 172.21.15.117:59942) 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f335889d640 1 -- 172.21.15.117:0/2769410192 learned_addr learned my addr 172.21.15.117:0/2769410192 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0c8030 con 0x7f334c02d6e0 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f335a0a0640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f335a0a0640 10 monclient(hunting): authenticate will time out at 1080.088076s 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter ms_handle_connect 0x7f334c0389e0 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter ms_handle_connect 0x7f334c02d6e0 2024-03-29T11:23:31.761 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter ms_handle_connect 0x7f334c030660 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809857205 0 0) 0x7f334c02dd10 con 0x7f334c0389e0 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.779+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3338004eb0 con 0x7f334c0389e0 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283961103 0 0) 0x7f334c02eba0 con 0x7f334c030660 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.780+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c02dd10 con 0x7f334c030660 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625727623 0 0) 0x7f334c0c8030 con 0x7f334c02d6e0 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c02eba0 con 0x7f334c02d6e0 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1376064278 0 0) 0x7f3338004eb0 con 0x7f334c0389e0 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f334c0c8030 con 0x7f334c0389e0 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3344003f60 con 0x7f334c0389e0 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:31.765 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1541576721 0 0) 0x7f334c02dd10 con 0x7f334c030660 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3338004eb0 con 0x7f334c030660 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3350001b70 con 0x7f334c030660 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.781+0000 7f3349ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3275323983 0 0) 0x7f334c0c8030 con 0x7f334c0389e0 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 >> v1:172.21.15.117:6790/0 conn(0x7f334c02d6e0 legacy=0x7f334c168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 >> v1:172.21.15.117:6789/0 conn(0x7f334c030660 legacy=0x7f334c02fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:31.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 monclient: found mon.b 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c02d150 con 0x7f334c0389e0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f335a0a0640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f335a0a0640 10 monclient: _renew_subs 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f335a0a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c0c84e0 con 0x7f334c0389e0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3344004470 con 0x7f334c0389e0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f3344001070 con 0x7f334c0389e0 2024-03-29T11:23:31.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f3349ffb640 10 monclient: dump: 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.782+0000 7f334affd640 4 set_mon_vals no callback set 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.783+0000 7f335a0a0640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:31.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.783+0000 7f335a0a0640 10 monclient: _renew_subs 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.783+0000 7f335a0a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.783+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c0c0b10 con 0x7f334c0389e0 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.783+0000 7f335a0a0640 1 librados: init done 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.785+0000 7f335a0a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.785+0000 7f335a0a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.785+0000 7f335a0a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:31.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.785+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c007b90 con 0x7f334c0389e0 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.785+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f33440011d0 con 0x7f334c0389e0 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.786+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f3344084940 con 0x7f334c0389e0 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.786+0000 7f3349ffb640 10 client.4402.objecter ms_dispatch 0x7f3354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.786+0000 7f3349ffb640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.786+0000 7f3349ffb640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.786+0000 7f3349ffb640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:23:31.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.787+0000 7f335909e640 10 monclient: get_auth_request con 0x7f333806de80 auth_method 0 2024-03-29T11:23:31.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.787+0000 7f3349ffb640 10 client.4402.objecter ms_handle_connect 0x7f333806de80 2024-03-29T11:23:31.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.793+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3344050220 con 0x7f334c0389e0 2024-03-29T11:23:31.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.793+0000 7f3349ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:31.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:31.793+0000 7f3349ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:32.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.164+0000 7f335a0a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.164+0000 7f335a0a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.164+0000 7f335a0a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:23:32.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.164+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f334c0c7e90 con 0x7f334c0389e0 2024-03-29T11:23:32.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.166+0000 7f3349ffb640 1 -- 172.21.15.117:0/2769410192 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f334c0c7e90 con 0x7f334c0389e0 2024-03-29T11:23:32.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.166+0000 7f3349ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:32.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.166+0000 7f3349ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:32.145 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:32.145 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.169+0000 7f335a0a0640 10 librados: watch_flush enter 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.169+0000 7f335a0a0640 10 librados: watch_flush exit 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.169+0000 7f335a0a0640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.169+0000 7f335a0a0640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 >> v1:172.21.15.117:6816/4276038175 conn(0x7f333806de80 legacy=0x7f3338070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 10 monclient: shutdown 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:32.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 >> v1:172.21.15.173:6789/0 conn(0x7f334c0389e0 legacy=0x7f334c0c0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 shutdown_connections 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 >> 172.21.15.117:0/2769410192 conn(0x7f334c005c00 msgr2=0x7f334c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 shutdown_connections 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 -- 172.21.15.117:0/2769410192 wait complete. 2024-03-29T11:23:32.150 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:32.170+0000 7f335a0a0640 1 librados: shutdown 2024-03-29T11:23:32.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:38.972 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:39.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.132+0000 7f4bc24ae640 10 monclient: get_monmap_and_config 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.132+0000 7f4bc24ae640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.132+0000 7f4bc24ae640 10 monclient: monmap: 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:39.132846+0000 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:39.132846+0000 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:39.112 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:39.113 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.133+0000 7f4bc24ae640 1 Processor -- start 2024-03-29T11:23:39.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 1 -- start start 2024-03-29T11:23:39.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient: init 2024-03-29T11:23:39.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc0b0500 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc0b08e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.134+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc0aeca0 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bc24ae640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc140620 con 0x7f4bbc0b08e0 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bc24ae640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc13daf0 con 0x7f4bbc0b0500 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bc24ae640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0ae180 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bc24ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 0x7f4bbc13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:57428/0 (socket says 172.21.15.117:57428) 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 -- 172.21.15.117:0/1985299202 learned_addr learned my addr 172.21.15.117:0/1985299202 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bc24ae640 10 monclient(hunting): authenticate will time out at 1087.443157s 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 -- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 legacy=0x7f4bbc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:23:39.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 -- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 legacy=0x7f4bbc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 --1- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 0x7f4bbc13d3e0 :-1 s=OPENED pgs=452143 cs=1 l=1).handle_message read tag failed 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbb7fe640 1 --1- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 0x7f4bbc13d3e0 :-1 s=OPENED pgs=452143 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.135+0000 7f4bbaffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.117:6789/0 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023987871 0 0) 0x7f4bbc0ae180 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:23:39.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.136+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4003520 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.137+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080499237 0 0) 0x7f4ba4003520 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.137+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 >> v1:172.21.15.173:6789/0 conn(0x7f4bbc0b0500 legacy=0x7f4bbc13ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0b08e0 legacy=0x7f4bbc13d3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 monclient: found mon.noname-b 2024-03-29T11:23:39.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0accf0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4bb4001a10 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: dump: 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.118 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bc24ae640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bc24ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bb4001bf0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 1 -- 172.21.15.117:0/1985299202 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4bb4002040 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bbaffd640 10 monclient: dump: 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bc24ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:39.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.138+0000 7f4bc24ae640 4 set_mon_vals no callback set 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.139+0000 7f4bc24ae640 10 monclient: shutdown 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 1 -- 172.21.15.117:0/1985299202 >> v1:172.21.15.117:6790/0 conn(0x7f4bbc0aeca0 legacy=0x7f4bbc14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 1 -- 172.21.15.117:0/1985299202 shutdown_connections 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 1 -- 172.21.15.117:0/1985299202 >> 172.21.15.117:0/1985299202 conn(0x7f4bbc06b660 msgr2=0x7f4bbc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 1 -- 172.21.15.117:0/1985299202 shutdown_connections 2024-03-29T11:23:39.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.140+0000 7f4bc24ae640 1 -- 172.21.15.117:0/1985299202 wait complete. 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.141+0000 7f4bc24ae640 10 monclient: build_initial_monmap 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.141+0000 7f4bc24ae640 10 monclient: monmap: 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.141+0000 7f4bc24ae640 1 librados: starting msgr at 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.141+0000 7f4bc24ae640 1 librados: starting objecter 2024-03-29T11:23:39.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.142+0000 7f4bc24ae640 1 Processor -- start 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.142+0000 7f4bc24ae640 1 -- start start 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.142+0000 7f4bc24ae640 1 librados: setting wanted keys 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.142+0000 7f4bc24ae640 1 librados: calling monclient init 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.142+0000 7f4bc24ae640 10 monclient: init 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bc24ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bc24ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-a con 0x7f4bbc0aeca0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bbbfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0aeca0 0x7f4bbc120a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:57910/0 (socket says 172.21.15.117:57910) 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bbbfff640 1 -- 172.21.15.117:0/3585313535 learned_addr learned my addr 172.21.15.117:0/3585313535 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bb8ff9640 10 client.?.objecter ms_handle_connect 0x7f4bbc0aeca0 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.143+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc0b0500 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc1d8960 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:39.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 client.?.objecter ms_handle_connect 0x7f4bbc0b0500 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc1d84d0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc122090 con 0x7f4bbc1d8960 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc11d050 con 0x7f4bbc0b0500 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145569693 0 0) 0x7f4bbc1d84d0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bc24ae640 10 monclient(hunting): authenticate will time out at 1087.452752s 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:39.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.144+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8004eb0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273559691 0 0) 0x7f4bbc11d050 con 0x7f4bbc0b0500 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bbc1d84d0 con 0x7f4bbc0b0500 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2135531425 0 0) 0x7f4ba8004eb0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba8005490 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4bac001870 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655080400 0 0) 0x7f4bbc1d84d0 con 0x7f4bbc0b0500 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba8004eb0 con 0x7f4bbc0b0500 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4bb00017f0 con 0x7f4bbc0b0500 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:39.135 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849093321 0 0) 0x7f4ba8005490 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 >> v1:172.21.15.117:6790/0 conn(0x7f4bbc0b0500 legacy=0x7f4bbc11fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.145+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 >> v1:172.21.15.173:6789/0 conn(0x7f4bbc1d8960 legacy=0x7f4bbc1df9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 1 monclient: found mon.a 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc11d1e0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:39.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 10 monclient: _renew_subs 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc1d8df0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 10 monclient: _renew_subs 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc1d92d0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.146+0000 7f4bc24ae640 1 librados: init done 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bac001d80 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4bac004900 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bb8ff9640 10 monclient: dump: 2024-03-29T11:23:39.137 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bba7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.148+0000 7f4bc24ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.149+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4bac01a8d0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.149+0000 7f4bc24ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.149+0000 7f4bc24ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.149+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c005080 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f4bac0848e0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bb8ff9640 10 client.4380.objecter ms_dispatch 0x7f4bbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bb8ff9640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bb8ff9640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bb8ff9640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:23:39.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.150+0000 7f4bbb7fe640 10 monclient: get_auth_request con 0x7f4ba806e0a0 auth_method 0 2024-03-29T11:23:39.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.151+0000 7f4bb8ff9640 10 client.4380.objecter ms_handle_connect 0x7f4ba806e0a0 2024-03-29T11:23:39.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.156+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bac0501c0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.156+0000 7f4bb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:39.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.156+0000 7f4bb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:39.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.536+0000 7f4bc24ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.536+0000 7f4bc24ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.536+0000 7f4bc24ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:39.515 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.536+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b9c005330 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.538+0000 7f4bb8ff9640 1 -- 172.21.15.117:0/3585313535 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f4bac0520f0 con 0x7f4bbc0aeca0 2024-03-29T11:23:39.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.538+0000 7f4bb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:39.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.538+0000 7f4bb8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:39.517 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:39.517 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 10 librados: watch_flush enter 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 10 librados: watch_flush exit 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 >> v1:172.21.15.117:6816/4276038175 conn(0x7f4ba806e0a0 legacy=0x7f4ba8070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 10 monclient: shutdown 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:39.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 >> v1:172.21.15.117:6789/0 conn(0x7f4bbc0aeca0 legacy=0x7f4bbc120a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:39.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 shutdown_connections 2024-03-29T11:23:39.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.541+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 >> 172.21.15.117:0/3585313535 conn(0x7f4bbc06b660 msgr2=0x7f4bbc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:39.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.542+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 shutdown_connections 2024-03-29T11:23:39.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.542+0000 7f4bc24ae640 1 -- 172.21.15.117:0/3585313535 wait complete. 2024-03-29T11:23:39.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:39.542+0000 7f4bc24ae640 1 librados: shutdown 2024-03-29T11:23:39.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:46.341 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:46.490 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.509+0000 7f6ef9f7c640 10 monclient: get_monmap_and_config 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.509+0000 7f6ef9f7c640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.509+0000 7f6ef9f7c640 10 monclient: monmap: 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:46.510176+0000 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:46.510176+0000 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.510+0000 7f6ef9f7c640 1 Processor -- start 2024-03-29T11:23:46.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.510+0000 7f6ef9f7c640 1 -- start start 2024-03-29T11:23:46.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.510+0000 7f6ef9f7c640 10 monclient: init 2024-03-29T11:23:46.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:46.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef40ae250 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef413f970 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef4142800 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40afe00 con 0x7f6ef413f970 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef4142c10 con 0x7f6ef40ae250 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40afb20 con 0x7f6ef4142800 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.511+0000 7f6ef9f7c640 10 monclient(hunting): authenticate will time out at 1094.820911s 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.514+0000 7f6ef37fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6ef40ae250 0x7f6ef413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:41147/0 (socket says 172.21.15.117:41147) 2024-03-29T11:23:46.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.514+0000 7f6ef37fe640 1 -- 172.21.15.117:0/2019793366 learned_addr learned my addr 172.21.15.117:0/2019793366 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043218257 0 0) 0x7f6ef40afe00 con 0x7f6ef413f970 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.515+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc003520 con 0x7f6ef413f970 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310205016 0 0) 0x7f6ef40afb20 con 0x7f6ef4142800 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef40afe00 con 0x7f6ef4142800 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648488607 0 0) 0x7f6ef4142c10 con 0x7f6ef40ae250 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef40afb20 con 0x7f6ef40ae250 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3448435204 0 0) 0x7f6edc003520 con 0x7f6ef413f970 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 >> v1:172.21.15.117:6790/0 conn(0x7f6ef4142800 legacy=0x7f6ef414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 >> v1:172.21.15.173:6789/0 conn(0x7f6ef40ae250 legacy=0x7f6ef413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 monclient: found mon.noname-a 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef40aab80 con 0x7f6ef413f970 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6ee8001690 con 0x7f6ef413f970 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: dump: 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.498 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:46.499 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:46.499 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:46.499 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef9f7c640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:23:46.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef9f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:46.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ee8001870 con 0x7f6ef413f970 2024-03-29T11:23:46.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 1 -- 172.21.15.117:0/2019793366 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6ee8001690 con 0x7f6ef413f970 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef27fc640 10 monclient: dump: 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef9f7c640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:46.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.516+0000 7f6ef9f7c640 4 set_mon_vals no callback set 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.517+0000 7f6ef9f7c640 10 monclient: shutdown 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.517+0000 7f6ef9f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.517+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/2019793366 >> v1:172.21.15.117:6789/0 conn(0x7f6ef413f970 legacy=0x7f6ef41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.517+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/2019793366 shutdown_connections 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.517+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/2019793366 >> 172.21.15.117:0/2019793366 conn(0x7f6ef40b35e0 msgr2=0x7f6ef40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.518+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/2019793366 shutdown_connections 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.519+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/2019793366 wait complete. 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.519+0000 7f6ef9f7c640 10 monclient: build_initial_monmap 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.519+0000 7f6ef9f7c640 10 monclient: monmap: 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:46.502 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 librados: starting msgr at 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 librados: starting objecter 2024-03-29T11:23:46.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 Processor -- start 2024-03-29T11:23:46.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 -- start start 2024-03-29T11:23:46.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 librados: setting wanted keys 2024-03-29T11:23:46.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 1 librados: calling monclient init 2024-03-29T11:23:46.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.520+0000 7f6ef9f7c640 10 monclient: init 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.521+0000 7f6ef9f7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.521+0000 7f6ef9f7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.521+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef40ae250 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef413f970 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef4142800 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef413adb0 con 0x7f6ef413f970 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40ac650 con 0x7f6ef4142800 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef40ad160 con 0x7f6ef40ae250 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.522+0000 7f6ef9f7c640 10 monclient(hunting): authenticate will time out at 1094.831402s 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6ef37fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f6ef40ae250 0x7f6ef40ac8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34204/0 (socket says 172.21.15.117:34204) 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6ef37fe640 1 -- 172.21.15.117:0/3901987896 learned_addr learned my addr 172.21.15.117:0/3901987896 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 client.?.objecter ms_handle_connect 0x7f6ef40ae250 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:46.505 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942627247 0 0) 0x7f6ef40ad160 con 0x7f6ef40ae250 2024-03-29T11:23:46.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:46.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:46.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:46.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:23:46.506 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.524+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee0004eb0 con 0x7f6ef40ae250 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2197377398 0 0) 0x7f6ee0004eb0 con 0x7f6ef40ae250 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ef40ad160 con 0x7f6ef40ae250 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6ee40018b0 con 0x7f6ef40ae250 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 10 monclient(hunting): dump: 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:46.508 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2374118449 0 0) 0x7f6ef40ad160 con 0x7f6ef40ae250 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 >> v1:172.21.15.173:6789/0 conn(0x7f6ef4142800 legacy=0x7f6ef40ab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.525+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 >> v1:172.21.15.117:6789/0 conn(0x7f6ef413f970 legacy=0x7f6ef4133700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.526+0000 7f6edbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.526+0000 7f6edbfff640 1 monclient: found mon.c 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.526+0000 7f6edbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.526+0000 7f6edbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.526+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4141340 con 0x7f6ef40ae250 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ee4001d00 con 0x7f6ef40ae250 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6ee4001b10 con 0x7f6ef40ae250 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:46.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6ef1ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.527+0000 7f6edbfff640 10 monclient: dump: 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:46.510 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 10 monclient: _renew_subs 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef41415f0 con 0x7f6ef40ae250 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 10 monclient: _renew_subs 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:46.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef4133ff0 con 0x7f6ef40ae250 2024-03-29T11:23:46.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.530+0000 7f6ef9f7c640 1 librados: init done 2024-03-29T11:23:46.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.533+0000 7f6ef9f7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.533+0000 7f6ef9f7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.533+0000 7f6ef9f7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:46.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.533+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef4075f10 con 0x7f6ef40ae250 2024-03-29T11:23:46.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.536+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f6ee401aaa0 con 0x7f6ef40ae250 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f6ee4084b20 con 0x7f6ef40ae250 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 10 client.4436.objecter ms_dispatch 0x7f6ef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6ef2ffd640 10 monclient: get_auth_request con 0x7f6ee006dc50 auth_method 0 2024-03-29T11:23:46.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.537+0000 7f6edbfff640 10 client.4436.objecter ms_handle_connect 0x7f6ee006dc50 2024-03-29T11:23:46.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.544+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ee4050230 con 0x7f6ef40ae250 2024-03-29T11:23:46.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.544+0000 7f6edbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:46.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.544+0000 7f6edbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6ef9f7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6ef9f7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6ef9f7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ef40ad2f0 con 0x7f6ef40ae250 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6edbfff640 1 -- 172.21.15.117:0/3901987896 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f6ee4052330 con 0x7f6ef40ae250 2024-03-29T11:23:46.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6edbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:46.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.877+0000 7f6edbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:46.858 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:46.858 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:46.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.881+0000 7f6ef9f7c640 10 librados: watch_flush enter 2024-03-29T11:23:46.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 10 librados: watch_flush exit 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 >> v1:172.21.15.117:6816/4276038175 conn(0x7f6ee006dc50 legacy=0x7f6ee0070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 10 monclient: shutdown 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 >> v1:172.21.15.117:6790/0 conn(0x7f6ef40ae250 legacy=0x7f6ef40ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 shutdown_connections 2024-03-29T11:23:46.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.883+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 >> 172.21.15.117:0/3901987896 conn(0x7f6ef40b35e0 msgr2=0x7f6ef40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:46.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.885+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 shutdown_connections 2024-03-29T11:23:46.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.885+0000 7f6ef9f7c640 1 -- 172.21.15.117:0/3901987896 wait complete. 2024-03-29T11:23:46.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:46.885+0000 7f6ef9f7c640 1 librados: shutdown 2024-03-29T11:23:46.883 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:53.690 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.866+0000 7fe5da896640 10 monclient: get_monmap_and_config 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.866+0000 7fe5da896640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.866+0000 7fe5da896640 10 monclient: monmap: 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:23:53.867635+0000 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:23:53.867635+0000 2024-03-29T11:23:53.847 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.848 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.848 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:53.848 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:23:53.848 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:23:53.848 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.867+0000 7fe5da896640 1 Processor -- start 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.867+0000 7fe5da896640 1 -- start start 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.867+0000 7fe5da896640 10 monclient: init 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.868+0000 7fe5da896640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.868+0000 7fe5da896640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:53.849 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.868+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d40a9fe0 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.868+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d40af600 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d40ad340 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40aaec0 con 0x7fe5d40af600 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40ab6a0 con 0x7fe5d40ad340 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40b00a0 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5da896640 10 monclient(hunting): authenticate will time out at 1102.178588s 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5d9894640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe5d40a9fe0 0x7fe5d413b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46346/0 (socket says 172.21.15.117:46346) 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.869+0000 7fe5d9894640 1 -- 172.21.15.117:0/768793147 learned_addr learned my addr 172.21.15.117:0/768793147 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:53.855 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532215017 0 0) 0x7fe5d40b00a0 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c0003520 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886989688 0 0) 0x7fe5d40aaec0 con 0x7fe5d40af600 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d40b00a0 con 0x7fe5d40af600 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 936356287 0 0) 0x7fe5c0003520 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.856 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 >> v1:172.21.15.173:6789/0 conn(0x7fe5d40ad340 legacy=0x7fe5d4149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 >> v1:172.21.15.117:6789/0 conn(0x7fe5d40af600 legacy=0x7fe5d40acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 monclient: found mon.noname-b 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d413beb0 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d0001a10 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5d8892640 10 monclient: dump: 2024-03-29T11:23:53.857 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5da896640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.872+0000 7fe5da896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5d0001b70 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 1 -- 172.21.15.117:0/768793147 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5d0001f80 con 0x7fe5d40a9fe0 2024-03-29T11:23:53.858 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5da896640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5da896640 4 set_mon_vals no callback set 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.873+0000 7fe5d8892640 10 monclient: dump: 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.859 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.874+0000 7fe5da896640 10 monclient: shutdown 2024-03-29T11:23:53.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.874+0000 7fe5da896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:53.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.874+0000 7fe5da896640 1 -- 172.21.15.117:0/768793147 >> v1:172.21.15.117:6790/0 conn(0x7fe5d40a9fe0 legacy=0x7fe5d413b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.874+0000 7fe5da896640 1 -- 172.21.15.117:0/768793147 shutdown_connections 2024-03-29T11:23:53.860 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.874+0000 7fe5da896640 1 -- 172.21.15.117:0/768793147 >> 172.21.15.117:0/768793147 conn(0x7fe5d40673e0 msgr2=0x7fe5d40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.877+0000 7fe5da896640 1 -- 172.21.15.117:0/768793147 shutdown_connections 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.877+0000 7fe5da896640 1 -- 172.21.15.117:0/768793147 wait complete. 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.879+0000 7fe5da896640 10 monclient: build_initial_monmap 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.879+0000 7fe5da896640 10 monclient: monmap: 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.861 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.880+0000 7fe5da896640 1 librados: starting msgr at 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.880+0000 7fe5da896640 1 librados: starting objecter 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.880+0000 7fe5da896640 1 Processor -- start 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.882+0000 7fe5da896640 1 -- start start 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.882+0000 7fe5da896640 1 librados: setting wanted keys 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.882+0000 7fe5da896640 1 librados: calling monclient init 2024-03-29T11:23:53.862 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.882+0000 7fe5da896640 10 monclient: init 2024-03-29T11:23:53.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:53.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:53.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-b con 0x7fe5d40ad340 addr v1:172.21.15.173:6789/0 2024-03-29T11:23:53.863 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-c con 0x7fe5d40af600 addr v1:172.21.15.117:6790/0 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient(hunting): picked mon.noname-a con 0x7fe5d413c9b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d406f7a0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d41d63c0 con 0x7fe5d40ad340 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d41d6ff0 con 0x7fe5d40af600 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5da896640 10 monclient(hunting): authenticate will time out at 1102.192847s 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5d9093640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe5d40af600 0x7fe5d4132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46536/0 (socket says 172.21.15.117:46536) 2024-03-29T11:23:53.864 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.883+0000 7fe5d9093640 1 -- 172.21.15.117:0/1636902089 learned_addr learned my addr 172.21.15.117:0/1636902089 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:23:53.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.885+0000 7fe5ca7fc640 10 client.?.objecter ms_handle_connect 0x7fe5d40af600 2024-03-29T11:23:53.865 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.885+0000 7fe5ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.886+0000 7fe5ca7fc640 10 client.?.objecter ms_handle_connect 0x7fe5d413c9b0 2024-03-29T11:23:53.866 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.886+0000 7fe5ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358112996 0 0) 0x7fe5d41d6ff0 con 0x7fe5d40af600 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5b8004eb0 con 0x7fe5d40af600 2024-03-29T11:23:53.869 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069766681 0 0) 0x7fe5d406f7a0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.887+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d41d6ff0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2157625436 0 0) 0x7fe5d41d6ff0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5d406f7a0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5cc001510 con 0x7fe5d413c9b0 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:53.870 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825940603 0 0) 0x7fe5d406f7a0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 >> v1:172.21.15.117:6790/0 conn(0x7fe5d40af600 legacy=0x7fe5d4132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 >> v1:172.21.15.173:6789/0 conn(0x7fe5d40ad340 legacy=0x7fe5d41d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 monclient: found mon.a 2024-03-29T11:23:53.871 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d41d7180 con 0x7fe5d413c9b0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 10 monclient: _renew_subs 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d406e6f0 con 0x7fe5d413c9b0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 10 monclient: _renew_subs 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d413f520 con 0x7fe5d413c9b0 2024-03-29T11:23:53.872 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.888+0000 7fe5da896640 1 librados: init done 2024-03-29T11:23:53.873 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.891+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5cc001a20 con 0x7fe5d413c9b0 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.891+0000 7fe5ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.891+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe5cc001d70 con 0x7fe5d413c9b0 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5ca7fc640 10 monclient: dump: 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:23:53.874 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5cb7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fe5cc019700 con 0x7fe5d413c9b0 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.892+0000 7fe5da896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.893+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fe5cc083720 con 0x7fe5d413c9b0 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.893+0000 7fe5ca7fc640 10 client.4389.objecter ms_dispatch 0x7fe5d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.893+0000 7fe5ca7fc640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.893+0000 7fe5ca7fc640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:53.875 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.893+0000 7fe5ca7fc640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T11:23:53.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.895+0000 7fe5da896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.895+0000 7fe5da896640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:53.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.895+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc005080 con 0x7fe5d413c9b0 2024-03-29T11:23:53.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.896+0000 7fe5d9894640 10 monclient: get_auth_request con 0x7fe5b806dd70 auth_method 0 2024-03-29T11:23:53.876 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.896+0000 7fe5ca7fc640 10 client.4389.objecter ms_handle_connect 0x7fe5b806dd70 2024-03-29T11:23:53.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.904+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5cc04f180 con 0x7fe5d413c9b0 2024-03-29T11:23:53.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.904+0000 7fe5ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:53.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:53.904+0000 7fe5ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:54.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.222+0000 7fe5da896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.222+0000 7fe5da896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.222+0000 7fe5da896640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:23:54.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.222+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5bc005330 con 0x7fe5d413c9b0 2024-03-29T11:23:54.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.223+0000 7fe5ca7fc640 1 -- 172.21.15.117:0/1636902089 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fe5cc0510b0 con 0x7fe5d413c9b0 2024-03-29T11:23:54.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.223+0000 7fe5ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:54.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.223+0000 7fe5ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:54.204 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:23:54.204 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:54.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.226+0000 7fe5da896640 10 librados: watch_flush enter 2024-03-29T11:23:54.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 10 librados: watch_flush exit 2024-03-29T11:23:54.208 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T11:23:54.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 10 client.4389.objecter successfully canceled tick 2024-03-29T11:23:54.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 >> v1:172.21.15.117:6816/4276038175 conn(0x7fe5b806dd70 legacy=0x7fe5b8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 10 monclient: shutdown 2024-03-29T11:23:54.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:54.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.228+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 >> v1:172.21.15.117:6789/0 conn(0x7fe5d413c9b0 legacy=0x7fe5d41de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:54.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.229+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 shutdown_connections 2024-03-29T11:23:54.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.229+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 >> 172.21.15.117:0/1636902089 conn(0x7fe5d40673e0 msgr2=0x7fe5d4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:54.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.230+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 shutdown_connections 2024-03-29T11:23:54.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.232+0000 7fe5da896640 1 -- 172.21.15.117:0/1636902089 wait complete. 2024-03-29T11:23:54.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:23:54.232+0000 7fe5da896640 1 librados: shutdown 2024-03-29T11:23:54.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:01.037 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.197+0000 7faf8614d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.197+0000 7faf8614d640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.197+0000 7faf8614d640 10 monclient: monmap: 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:01.198478+0000 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:01.198478+0000 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:01.178 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:01.179 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:01.179 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.198+0000 7faf8614d640 1 Processor -- start 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.198+0000 7faf8614d640 1 -- start start 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.198+0000 7faf8614d640 10 monclient: init 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-a con 0x7faf801457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-c con 0x7faf8013f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:01.180 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-b con 0x7faf800a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800aaef0 con 0x7faf801457f0 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf80145bd0 con 0x7faf8013f650 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf80145eb0 con 0x7faf800a9d70 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf8614d640 10 monclient(hunting): authenticate will time out at 1109.508424s 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf7ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7faf800a9d70 0x7faf8014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:33512/0 (socket says 172.21.15.117:33512) 2024-03-29T11:24:01.181 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.199+0000 7faf7ffff640 1 -- 172.21.15.117:0/967521098 learned_addr learned my addr 172.21.15.117:0/967521098 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977646234 0 0) 0x7faf80145eb0 con 0x7faf800a9d70 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:24:01.182 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.201+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf68003520 con 0x7faf800a9d70 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768044577 0 0) 0x7faf800aaef0 con 0x7faf801457f0 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf80145eb0 con 0x7faf801457f0 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234470418 0 0) 0x7faf68003520 con 0x7faf800a9d70 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 >> v1:172.21.15.173:6789/0 conn(0x7faf8013f650 legacy=0x7faf800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 >> v1:172.21.15.117:6789/0 conn(0x7faf801457f0 legacy=0x7faf8013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:01.183 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80146340 con 0x7faf800a9d70 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7faf78001a10 con 0x7faf800a9d70 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf7e7fc640 10 monclient: dump: 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.184 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf8614d640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:24:01.185 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.202+0000 7faf8614d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf78001b70 con 0x7faf800a9d70 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 1 -- 172.21.15.117:0/967521098 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7faf78001f80 con 0x7faf800a9d70 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf7e7fc640 10 monclient: dump: 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.187 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf8614d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf8614d640 4 set_mon_vals no callback set 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.203+0000 7faf8614d640 10 monclient: shutdown 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 1 -- 172.21.15.117:0/967521098 >> v1:172.21.15.117:6790/0 conn(0x7faf800a9d70 legacy=0x7faf8014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 1 -- 172.21.15.117:0/967521098 shutdown_connections 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 1 -- 172.21.15.117:0/967521098 >> 172.21.15.117:0/967521098 conn(0x7faf80066a00 msgr2=0x7faf80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 1 -- 172.21.15.117:0/967521098 shutdown_connections 2024-03-29T11:24:01.188 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.204+0000 7faf8614d640 1 -- 172.21.15.117:0/967521098 wait complete. 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.205+0000 7faf8614d640 10 monclient: build_initial_monmap 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.205+0000 7faf8614d640 10 monclient: monmap: 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.206+0000 7faf8614d640 1 librados: starting msgr at 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.206+0000 7faf8614d640 1 librados: starting objecter 2024-03-29T11:24:01.189 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.206+0000 7faf8614d640 1 Processor -- start 2024-03-29T11:24:01.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.206+0000 7faf8614d640 1 -- start start 2024-03-29T11:24:01.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.206+0000 7faf8614d640 1 librados: setting wanted keys 2024-03-29T11:24:01.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.207+0000 7faf8614d640 1 librados: calling monclient init 2024-03-29T11:24:01.190 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.207+0000 7faf8614d640 10 monclient: init 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-c con 0x7faf800a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-b con 0x7faf8013f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient(hunting): picked mon.noname-a con 0x7faf801457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800b12e0 con 0x7faf801457f0 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf801dbbb0 con 0x7faf8013f650 2024-03-29T11:24:01.195 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf800b1570 con 0x7faf800a9d70 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf8614d640 10 monclient(hunting): authenticate will time out at 1109.517134s 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf7f7fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7faf800a9d70 0x7faf80144540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34100/0 (socket says 172.21.15.117:34100) 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf7f7fe640 1 -- 172.21.15.117:0/395855150 learned_addr learned my addr 172.21.15.117:0/395855150 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 client.?.objecter ms_handle_connect 0x7faf800a9d70 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514605116 0 0) 0x7faf800b1570 con 0x7faf800a9d70 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c004eb0 con 0x7faf800a9d70 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2390813525 0 0) 0x7faf6c004eb0 con 0x7faf800a9d70 2024-03-29T11:24:01.196 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf800b1570 con 0x7faf800a9d70 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7faf70001a20 con 0x7faf800a9d70 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): dump: 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.197 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 client.?.objecter ms_handle_connect 0x7faf8013f650 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.208+0000 7faf67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.209+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1971310894 0 0) 0x7faf800b1570 con 0x7faf800a9d70 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.209+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 >> v1:172.21.15.173:6789/0 conn(0x7faf8013f650 legacy=0x7faf8013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.209+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 >> v1:172.21.15.117:6789/0 conn(0x7faf801457f0 legacy=0x7faf801e30a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 1 monclient: found mon.c 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800b1700 con 0x7faf800a9d70 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:24:01.198 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 10 monclient: _renew_subs 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf80144c50 con 0x7faf800a9d70 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 10 monclient: _renew_subs 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf80140690 con 0x7faf800a9d70 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.211+0000 7faf8614d640 1 librados: init done 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.212+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf70001d00 con 0x7faf800a9d70 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.212+0000 7faf67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.212+0000 7faf7dffb640 4 set_mon_vals no callback set 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7faf70003a60 con 0x7faf800a9d70 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf67fff640 10 monclient: dump: 2024-03-29T11:24:01.199 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf8614d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf8614d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf8614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.200 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.213+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf60005080 con 0x7faf800a9d70 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.215+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7faf70003c80 con 0x7faf800a9d70 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7faf70084bc0 con 0x7faf800a9d70 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 10 client.4448.objecter ms_dispatch 0x7faf80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:01.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:24:01.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf7effd640 10 monclient: get_auth_request con 0x7faf6c06dba0 auth_method 0 2024-03-29T11:24:01.202 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.216+0000 7faf67fff640 10 client.4448.objecter ms_handle_connect 0x7faf6c06dba0 2024-03-29T11:24:01.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.223+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf70050500 con 0x7faf800a9d70 2024-03-29T11:24:01.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.223+0000 7faf67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:01.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.223+0000 7faf67fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:01.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.522+0000 7faf8614d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.502 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.522+0000 7faf8614d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.522+0000 7faf8614d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:01.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.522+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf60005330 con 0x7faf800a9d70 2024-03-29T11:24:01.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.523+0000 7faf67fff640 1 -- 172.21.15.117:0/395855150 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7faf70052430 con 0x7faf800a9d70 2024-03-29T11:24:01.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.523+0000 7faf67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:01.504 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.523+0000 7faf67fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:01.504 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:01.505 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:01.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.526+0000 7faf8614d640 10 librados: watch_flush enter 2024-03-29T11:24:01.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.526+0000 7faf8614d640 10 librados: watch_flush exit 2024-03-29T11:24:01.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.526+0000 7faf8614d640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:24:01.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.526+0000 7faf8614d640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:24:01.507 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 >> v1:172.21.15.117:6816/4276038175 conn(0x7faf6c06dba0 legacy=0x7faf6c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 10 monclient: shutdown 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 >> v1:172.21.15.117:6790/0 conn(0x7faf800a9d70 legacy=0x7faf80144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 shutdown_connections 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 >> 172.21.15.117:0/395855150 conn(0x7faf80066a00 msgr2=0x7faf800b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 shutdown_connections 2024-03-29T11:24:01.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 -- 172.21.15.117:0/395855150 wait complete. 2024-03-29T11:24:01.509 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:01.527+0000 7faf8614d640 1 librados: shutdown 2024-03-29T11:24:01.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:08.330 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:08.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.499+0000 7fd1e05b5640 10 monclient: get_monmap_and_config 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.499+0000 7fd1e05b5640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.499+0000 7fd1e05b5640 10 monclient: monmap: 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:08.500959+0000 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:08.500959+0000 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:08.481 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.500+0000 7fd1e05b5640 1 Processor -- start 2024-03-29T11:24:08.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.500+0000 7fd1e05b5640 1 -- start start 2024-03-29T11:24:08.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.500+0000 7fd1e05b5640 10 monclient: init 2024-03-29T11:24:08.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d813f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d80a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d80aaef0 con 0x7fd1d813f650 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8145bd0 con 0x7fd1d81457f0 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8145eb0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.501+0000 7fd1e05b5640 10 monclient(hunting): authenticate will time out at 1116.810956s 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.502+0000 7fd1deb52640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd1d80a9d70 0x7fd1d814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48420/0 (socket says 172.21.15.117:48420) 2024-03-29T11:24:08.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.502+0000 7fd1deb52640 1 -- 172.21.15.117:0/1672690336 learned_addr learned my addr 172.21.15.117:0/1672690336 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:08.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339944563 0 0) 0x7fd1d8145eb0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4003520 con 0x7fd1d80a9d70 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2883488587 0 0) 0x7fd1c4003520 con 0x7fd1d80a9d70 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.504+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 >> v1:172.21.15.173:6789/0 conn(0x7fd1d81457f0 legacy=0x7fd1d813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 >> v1:172.21.15.117:6789/0 conn(0x7fd1d813f650 legacy=0x7fd1d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 1 monclient: found mon.noname-b 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8146340 con 0x7fd1d80a9d70 2024-03-29T11:24:08.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd1d0001a10 con 0x7fd1d80a9d70 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.505+0000 7fd1dd34f640 10 monclient: dump: 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.488 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1e05b5640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1e05b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1d0001c70 con 0x7fd1d80a9d70 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1dd34f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1e05b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:08.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.506+0000 7fd1e05b5640 4 set_mon_vals no callback set 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.507+0000 7fd1dd34f640 1 -- 172.21.15.117:0/1672690336 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd1d0002080 con 0x7fd1d80a9d70 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.507+0000 7fd1dd34f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.507+0000 7fd1dd34f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.507+0000 7fd1dd34f640 10 monclient: dump: 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:08.491 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.508+0000 7fd1e05b5640 10 monclient: shutdown 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.508+0000 7fd1e05b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.508+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1672690336 >> v1:172.21.15.117:6790/0 conn(0x7fd1d80a9d70 legacy=0x7fd1d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.508+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1672690336 shutdown_connections 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.508+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1672690336 >> 172.21.15.117:0/1672690336 conn(0x7fd1d8066a00 msgr2=0x7fd1d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.509+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1672690336 shutdown_connections 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.509+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1672690336 wait complete. 2024-03-29T11:24:08.492 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.509+0000 7fd1e05b5640 10 monclient: build_initial_monmap 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 10 monclient: monmap: 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 librados: starting msgr at 2024-03-29T11:24:08.493 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 librados: starting objecter 2024-03-29T11:24:08.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 Processor -- start 2024-03-29T11:24:08.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 -- start start 2024-03-29T11:24:08.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 librados: setting wanted keys 2024-03-29T11:24:08.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 1 librados: calling monclient init 2024-03-29T11:24:08.494 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.510+0000 7fd1e05b5640 10 monclient: init 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d80a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d813f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8141a10 con 0x7fd1d813f650 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8141c80 con 0x7fd1d81457f0 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d8144e80 con 0x7fd1d80a9d70 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1e05b5640 10 monclient(hunting): authenticate will time out at 1116.820503s 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.511+0000 7fd1de351640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fd1d80a9d70 0x7fd1d81da780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48918/0 (socket says 172.21.15.117:48918) 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1de351640 1 -- 172.21.15.117:0/1620464850 learned_addr learned my addr 172.21.15.117:0/1620464850 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 client.?.objecter ms_handle_connect 0x7fd1d80a9d70 2024-03-29T11:24:08.495 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140193962 0 0) 0x7fd1d8144e80 con 0x7fd1d80a9d70 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.512+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc004eb0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 108387308 0 0) 0x7fd1bc004eb0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1bc005300 con 0x7fd1d80a9d70 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd1d4001790 con 0x7fd1d80a9d70 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 10 monclient(hunting): dump: 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.496 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4197624479 0 0) 0x7fd1bc005300 con 0x7fd1d80a9d70 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 >> v1:172.21.15.173:6789/0 conn(0x7fd1d81457f0 legacy=0x7fd1d81427b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:08.497 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.513+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 >> v1:172.21.15.117:6789/0 conn(0x7fd1d813f650 legacy=0x7fd1d81420a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 1 monclient: found mon.c 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d81dae90 con 0x7fd1d80a9d70 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:08.499 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.515+0000 7fd1ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.516+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1d4001d60 con 0x7fd1d80a9d70 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.516+0000 7fd1ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.516+0000 7fd1cffff640 4 set_mon_vals no callback set 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd1d4001790 con 0x7fd1d80a9d70 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1ceffd640 10 monclient: dump: 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:08.500 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1e05b5640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1e05b5640 10 monclient: _renew_subs 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1e05b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.517+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d81db0e0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.518+0000 7fd1e05b5640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.518+0000 7fd1e05b5640 10 monclient: _renew_subs 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.518+0000 7fd1e05b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.518+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d8139f80 con 0x7fd1d80a9d70 2024-03-29T11:24:08.501 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.518+0000 7fd1e05b5640 1 librados: init done 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.519+0000 7fd1e05b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.519+0000 7fd1e05b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.519+0000 7fd1e05b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.519+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d8144c70 con 0x7fd1d80a9d70 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.520+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd1d401a810 con 0x7fd1d80a9d70 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ddb50640 10 monclient: get_auth_request con 0x7fd1bc06e070 auth_method 0 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fd1d4054580 con 0x7fd1d80a9d70 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 10 client.4454.objecter ms_dispatch 0x7fd1d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:24:08.503 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.521+0000 7fd1ceffd640 10 client.4454.objecter ms_handle_connect 0x7fd1bc06e070 2024-03-29T11:24:08.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.530+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1d4050820 con 0x7fd1d80a9d70 2024-03-29T11:24:08.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.530+0000 7fd1ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:08.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.530+0000 7fd1ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:08.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.846+0000 7fd1e05b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.846+0000 7fd1e05b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.846+0000 7fd1e05b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:08.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.846+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d8072080 con 0x7fd1d80a9d70 2024-03-29T11:24:08.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.848+0000 7fd1ceffd640 1 -- 172.21.15.117:0/1620464850 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fd1d40509c0 con 0x7fd1d80a9d70 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.848+0000 7fd1ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:08.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.848+0000 7fd1ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:08.832 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:08.834 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.853+0000 7fd1e05b5640 10 librados: watch_flush enter 2024-03-29T11:24:08.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.854+0000 7fd1e05b5640 10 librados: watch_flush exit 2024-03-29T11:24:08.836 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.854+0000 7fd1e05b5640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.854+0000 7fd1e05b5640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd1bc06e070 legacy=0x7fd1bc070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 10 monclient: shutdown 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 >> v1:172.21.15.117:6790/0 conn(0x7fd1d80a9d70 legacy=0x7fd1d81da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 shutdown_connections 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 >> 172.21.15.117:0/1620464850 conn(0x7fd1d8066a00 msgr2=0x7fd1d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 shutdown_connections 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 -- 172.21.15.117:0/1620464850 wait complete. 2024-03-29T11:24:08.837 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:08.855+0000 7fd1e05b5640 1 librados: shutdown 2024-03-29T11:24:08.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:15.660 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.822+0000 7f0d1b1a7640 10 monclient: get_monmap_and_config 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.822+0000 7f0d1b1a7640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.822+0000 7f0d1b1a7640 10 monclient: monmap: 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:15.823382+0000 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:15.823382+0000 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:15.804 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:15.805 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:15.805 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.823+0000 7f0d1b1a7640 1 Processor -- start 2024-03-29T11:24:15.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.823+0000 7f0d1b1a7640 1 -- start start 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.823+0000 7f0d1b1a7640 10 monclient: init 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-c con 0x7f0d140b0860 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-a con 0x7f0d1413ef30 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-b con 0x7f0d140af8e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140b0c40 con 0x7f0d1413ef30 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140aed10 con 0x7f0d140b0860 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140afd50 con 0x7f0d140af8e0 2024-03-29T11:24:15.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.824+0000 7f0d1b1a7640 10 monclient(hunting): authenticate will time out at 1124.133514s 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d19744640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f0d140af8e0 0x7f0d14149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:57688/0 (socket says 172.21.15.117:57688) 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d19744640 1 -- 172.21.15.117:0/352087657 learned_addr learned my addr 172.21.15.117:0/352087657 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254233372 0 0) 0x7f0d140afd50 con 0x7f0d140af8e0 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:24:15.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.825+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc003520 con 0x7f0d140af8e0 2024-03-29T11:24:15.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.828+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116151739 0 0) 0x7f0cfc003520 con 0x7f0d140af8e0 2024-03-29T11:24:15.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.828+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 >> v1:172.21.15.173:6789/0 conn(0x7f0d140b0860 legacy=0x7f0d1413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:15.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.828+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 >> v1:172.21.15.117:6789/0 conn(0x7f0d1413ef30 legacy=0x7f0d140af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:15.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140ab1b0 con 0x7f0d140af8e0 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d1b1a7640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d1b1a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0d0c001a10 con 0x7f0d140af8e0 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d137fe640 10 monclient: dump: 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.814 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.829+0000 7f0d1b1a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d0c001ca0 con 0x7f0d140af8e0 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 1 -- 172.21.15.117:0/352087657 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0d0c002030 con 0x7f0d140af8e0 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d137fe640 10 monclient: dump: 2024-03-29T11:24:15.815 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d1b1a7640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.830+0000 7f0d1b1a7640 4 set_mon_vals no callback set 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 10 monclient: shutdown 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/352087657 >> v1:172.21.15.117:6790/0 conn(0x7f0d140af8e0 legacy=0x7f0d14149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/352087657 shutdown_connections 2024-03-29T11:24:15.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/352087657 >> 172.21.15.117:0/352087657 conn(0x7f0d1406b660 msgr2=0x7f0d1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:15.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.831+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/352087657 shutdown_connections 2024-03-29T11:24:15.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.832+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/352087657 wait complete. 2024-03-29T11:24:15.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.832+0000 7f0d1b1a7640 10 monclient: build_initial_monmap 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.832+0000 7f0d1b1a7640 10 monclient: monmap: 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 librados: starting msgr at 2024-03-29T11:24:15.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 librados: starting objecter 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 Processor -- start 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 -- start start 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 librados: setting wanted keys 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 1 librados: calling monclient init 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.833+0000 7f0d1b1a7640 10 monclient: init 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-c con 0x7f0d140af8e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-b con 0x7f0d140b0860 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient(hunting): picked mon.noname-a con 0x7f0d1413ef30 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d141db760 con 0x7f0d1413ef30 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d14135170 con 0x7f0d140b0860 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d14136de0 con 0x7f0d140af8e0 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:15.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.834+0000 7f0d1b1a7640 10 monclient(hunting): authenticate will time out at 1124.143398s 2024-03-29T11:24:15.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.837+0000 7f0d13fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0d140b0860 0x7f0d14134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:45747/0 (socket says 172.21.15.117:45747) 2024-03-29T11:24:15.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.837+0000 7f0d13fff640 1 -- 172.21.15.117:0/4240737788 learned_addr learned my addr 172.21.15.117:0/4240737788 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.837+0000 7f0d117fa640 10 client.?.objecter ms_handle_connect 0x7f0d140b0860 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.837+0000 7f0d117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724412739 0 0) 0x7f0d14135170 con 0x7f0d140b0860 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d00004eb0 con 0x7f0d140b0860 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 client.?.objecter ms_handle_connect 0x7f0d140af8e0 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337445070 0 0) 0x7f0d14136de0 con 0x7f0d140af8e0 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:15.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14135170 con 0x7f0d140af8e0 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189184062 0 0) 0x7f0d00004eb0 con 0x7f0d140b0860 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d14136de0 con 0x7f0d140b0860 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0d080017d0 con 0x7f0d140b0860 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): dump: 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.823 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.839+0000 7f0d117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1843954183 0 0) 0x7f0d14135170 con 0x7f0d140af8e0 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d00004eb0 con 0x7f0d140af8e0 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0d04001510 con 0x7f0d140af8e0 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient(hunting): dump: 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.824 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2470613094 0 0) 0x7f0d14136de0 con 0x7f0d140b0860 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 >> v1:172.21.15.117:6790/0 conn(0x7f0d140af8e0 legacy=0x7f0d141dbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 >> v1:172.21.15.117:6789/0 conn(0x7f0d1413ef30 legacy=0x7f0d14134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 monclient: found mon.b 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:15.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140b4310 con 0x7f0d140b0860 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 10 monclient: _renew_subs 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d141dc300 con 0x7f0d140b0860 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 10 monclient: _renew_subs 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d140b3a50 con 0x7f0d140b0860 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d1b1a7640 1 librados: init done 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d08001930 con 0x7f0d140b0860 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:15.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0d08004a00 con 0x7f0d140b0860 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d117fa640 10 monclient: dump: 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:15.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.840+0000 7f0d12ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.842+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f0d08004b60 con 0x7f0d140b0860 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.843+0000 7f0d1b1a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.843+0000 7f0d1b1a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.843+0000 7f0d1b1a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.843+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf4005080 con 0x7f0d140b0860 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.843+0000 7f0d18f43640 10 monclient: get_auth_request con 0x7f0d0006df10 auth_method 0 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 10 client.4414.objecter ms_handle_connect 0x7f0d0006df10 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f0d080849e0 con 0x7f0d140b0860 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 10 client.4414.objecter ms_dispatch 0x7f0d14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:15.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.844+0000 7f0d117fa640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:24:15.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.852+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d080502c0 con 0x7f0d140b0860 2024-03-29T11:24:15.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.852+0000 7f0d117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:15.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:15.852+0000 7f0d117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:16.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.175+0000 7f0d1b1a7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.175+0000 7f0d1b1a7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.175+0000 7f0d1b1a7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:16.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.175+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf4005330 con 0x7f0d140b0860 2024-03-29T11:24:16.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.176+0000 7f0d117fa640 1 -- 172.21.15.117:0/4240737788 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f0d080521f0 con 0x7f0d140b0860 2024-03-29T11:24:16.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.176+0000 7f0d117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:16.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.176+0000 7f0d117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:16.158 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:16.158 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:16.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.180+0000 7f0d1b1a7640 10 librados: watch_flush enter 2024-03-29T11:24:16.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.180+0000 7f0d1b1a7640 10 librados: watch_flush exit 2024-03-29T11:24:16.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.180+0000 7f0d1b1a7640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:24:16.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.180+0000 7f0d1b1a7640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:24:16.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 >> v1:172.21.15.117:6816/4276038175 conn(0x7f0d0006df10 legacy=0x7f0d000703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 10 monclient: shutdown 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 >> v1:172.21.15.173:6789/0 conn(0x7f0d140b0860 legacy=0x7f0d14134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 shutdown_connections 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 >> 172.21.15.117:0/4240737788 conn(0x7f0d1406b660 msgr2=0x7f0d1413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 shutdown_connections 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 -- 172.21.15.117:0/4240737788 wait complete. 2024-03-29T11:24:16.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:16.181+0000 7f0d1b1a7640 1 librados: shutdown 2024-03-29T11:24:16.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:22.985 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:23.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.159+0000 7fc53fcb3640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.159+0000 7fc53fcb3640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.159+0000 7fc53fcb3640 10 monclient: monmap: 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:23.160106+0000 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:23.160106+0000 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:23.142 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.161+0000 7fc53fcb3640 1 Processor -- start 2024-03-29T11:24:23.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.161+0000 7fc53fcb3640 1 -- start start 2024-03-29T11:24:23.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.161+0000 7fc53fcb3640 10 monclient: init 2024-03-29T11:24:23.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.162+0000 7fc53fcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.162+0000 7fc53fcb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:23.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.162+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-b con 0x7fc5380a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:23.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.165+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-a con 0x7fc53813ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:23.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.165+0000 7fc53d24e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 0x7fc53813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52009/0 (socket says 172.21.15.117:52009) 2024-03-29T11:24:23.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.165+0000 7fc53d24e640 1 -- 172.21.15.117:0/3339155702 learned_addr learned my addr 172.21.15.117:0/3339155702 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.166+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5380af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53d24e640 1 -- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 legacy=0x7fc53813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53d24e640 1 -- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 legacy=0x7fc53813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53d24e640 1 --1- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 0x7fc53813b5f0 :-1 s=OPENED pgs=496713 cs=1 l=1).handle_message read tag failed 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53d24e640 1 --1- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 0x7fc53813b5f0 :-1 s=OPENED pgs=496713 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380afd10 con 0x7fc53813ee90 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380afd10 con 0x7fc5380af900 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380af6f0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53fcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.169+0000 7fc53fcb3640 10 monclient(hunting): authenticate will time out at 1131.478998s 2024-03-29T11:24:23.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.117:6789/0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36606380 0 0) 0x7fc5380af6f0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc524003520 con 0x7fc5380a9ac0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295457517 0 0) 0x7fc524003520 con 0x7fc5380a9ac0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 >> v1:172.21.15.173:6789/0 conn(0x7fc5380af900 legacy=0x7fc538149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 legacy=0x7fc53813b5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 monclient: found mon.noname-b 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5380aac80 con 0x7fc5380a9ac0 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc5340015c0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: dump: 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.157 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5340017e0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.170+0000 7fc53ca4d640 1 -- 172.21.15.117:0/3339155702 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc534001b70 con 0x7fc5380a9ac0 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53ca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53ca4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53ca4d640 10 monclient: dump: 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:23.158 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 4 set_mon_vals no callback set 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 10 monclient: shutdown 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.171+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 >> v1:172.21.15.117:6790/0 conn(0x7fc5380a9ac0 legacy=0x7fc5380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.173+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 shutdown_connections 2024-03-29T11:24:23.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.173+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 >> 172.21.15.117:0/3339155702 conn(0x7fc5380b35e0 msgr2=0x7fc5380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.173+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 shutdown_connections 2024-03-29T11:24:23.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.173+0000 7fc53fcb3640 1 -- 172.21.15.117:0/3339155702 wait complete. 2024-03-29T11:24:23.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.174+0000 7fc53fcb3640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.174+0000 7fc53fcb3640 10 monclient: monmap: 2024-03-29T11:24:23.162 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 librados: starting msgr at 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 librados: starting objecter 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 Processor -- start 2024-03-29T11:24:23.163 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 -- start start 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 librados: setting wanted keys 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 1 librados: calling monclient init 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.175+0000 7fc53fcb3640 10 monclient: init 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.176+0000 7fc53fcb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.176+0000 7fc53fcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.179+0000 7fc53da4f640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fc5380a9ac0 0x7fc538139780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:43799/0 (socket says 172.21.15.117:43799) 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53da4f640 1 -- 172.21.15.117:0/52076013 learned_addr learned my addr 172.21.15.117:0/52076013 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5380a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc52e7fc640 10 client.?.objecter ms_handle_connect 0x7fc5380a9ac0 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc52e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-b con 0x7fc5380af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53fcb3640 10 monclient(hunting): picked mon.noname-a con 0x7fc53813ee90 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5381c7790 con 0x7fc53813ee90 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.180+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5381c6100 con 0x7fc5380af900 2024-03-29T11:24:23.164 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.181+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5381c7270 con 0x7fc5380a9ac0 2024-03-29T11:24:23.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.181+0000 7fc53fcb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.181+0000 7fc53fcb3640 10 monclient(hunting): authenticate will time out at 1131.490233s 2024-03-29T11:24:23.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826668926 0 0) 0x7fc5381c7270 con 0x7fc5380a9ac0 2024-03-29T11:24:23.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.165 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.166 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.166 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:24:23.166 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51c004eb0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2979974379 0 0) 0x7fc51c004eb0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5381c7270 con 0x7fc5380a9ac0 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc534001580 con 0x7fc5380a9ac0 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.167 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:23.168 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.184+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040676519 0 0) 0x7fc5381c7270 con 0x7fc5380a9ac0 2024-03-29T11:24:23.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.185+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 >> v1:172.21.15.173:6789/0 conn(0x7fc5380af900 legacy=0x7fc5381d2e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:23.169 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.186+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 >> v1:172.21.15.117:6789/0 conn(0x7fc53813ee90 legacy=0x7fc5381de6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:23.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.171 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 1 monclient: found mon.c 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538139e90 con 0x7fc5380a9ac0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.187+0000 7fc52e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 10 monclient: _renew_subs 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53813a140 con 0x7fc5380a9ac0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 10 monclient: _renew_subs 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.172 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5381d3750 con 0x7fc5380a9ac0 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc53fcb3640 1 librados: init done 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5340038a0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc534003cf0 con 0x7fc5380a9ac0 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.188+0000 7fc52e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.189+0000 7fc52e7fc640 10 monclient: dump: 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:23.173 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:23.174 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:23.174 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:23.174 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.189+0000 7fc52ffff640 4 set_mon_vals no callback set 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.192+0000 7fc53fcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.192+0000 7fc53fcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.192+0000 7fc53fcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.192+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc520005080 con 0x7fc5380a9ac0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.192+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fc534003e50 con 0x7fc5380a9ac0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fc534084c70 con 0x7fc5380a9ac0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc53d24e640 10 monclient: get_auth_request con 0x7fc51c06dc90 auth_method 0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 10 client.4466.objecter ms_dispatch 0x7fc538000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.175 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.193+0000 7fc52e7fc640 10 client.4466.objecter ms_handle_connect 0x7fc51c06dc90 2024-03-29T11:24:23.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.203+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc534050550 con 0x7fc5380a9ac0 2024-03-29T11:24:23.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.203+0000 7fc52e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.184 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.203+0000 7fc52e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.554+0000 7fc53fcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.554+0000 7fc53fcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.554+0000 7fc53fcb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:23.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.554+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc520005330 con 0x7fc5380a9ac0 2024-03-29T11:24:23.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.555+0000 7fc52e7fc640 1 -- 172.21.15.117:0/52076013 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fc534052480 con 0x7fc5380a9ac0 2024-03-29T11:24:23.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.555+0000 7fc52e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:23.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.555+0000 7fc52e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.540 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:23.541 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:23.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 10 librados: watch_flush enter 2024-03-29T11:24:23.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 10 librados: watch_flush exit 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 >> v1:172.21.15.117:6816/4276038175 conn(0x7fc51c06dc90 legacy=0x7fc51c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 10 monclient: shutdown 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 >> v1:172.21.15.117:6790/0 conn(0x7fc5380a9ac0 legacy=0x7fc538139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 shutdown_connections 2024-03-29T11:24:23.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.559+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 >> 172.21.15.117:0/52076013 conn(0x7fc5380b35e0 msgr2=0x7fc53813b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.560+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 shutdown_connections 2024-03-29T11:24:23.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.560+0000 7fc53fcb3640 1 -- 172.21.15.117:0/52076013 wait complete. 2024-03-29T11:24:23.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:23.560+0000 7fc53fcb3640 1 librados: shutdown 2024-03-29T11:24:23.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:30.365 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:30.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.527+0000 7fdd06ef4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:30.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.527+0000 7fdd06ef4640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.508 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.527+0000 7fdd06ef4640 10 monclient: monmap: 2024-03-29T11:24:30.508 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:30.527832+0000 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:30.527832+0000 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:30.509 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.528+0000 7fdd06ef4640 1 Processor -- start 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.528+0000 7fdd06ef4640 1 -- start start 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.528+0000 7fdd06ef4640 10 monclient: init 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.529+0000 7fdd06ef4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.529+0000 7fdd06ef4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.529+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf8006a70 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:30.510 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.529+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf802b100 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd04c90640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdcf8006a70 0x7fdcf80c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59598/0 (socket says 172.21.15.117:59598) 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf8029c30 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf802a270 con 0x7fdcf802b100 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf802c440 con 0x7fdcf8029c30 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf80c9440 con 0x7fdcf8006a70 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd06ef4640 10 monclient(hunting): authenticate will time out at 1138.838858s 2024-03-29T11:24:30.511 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.530+0000 7fdd04c90640 1 -- 172.21.15.117:0/3794908956 learned_addr learned my addr 172.21.15.117:0/3794908956 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835315897 0 0) 0x7fdcf80c9440 con 0x7fdcf8006a70 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:24:30.512 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.531+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0003520 con 0x7fdcf8006a70 2024-03-29T11:24:30.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.532+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2759275792 0 0) 0x7fdcf0003520 con 0x7fdcf8006a70 2024-03-29T11:24:30.513 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.532+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 >> v1:172.21.15.173:6789/0 conn(0x7fdcf8029c30 legacy=0x7fdcf80d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:30.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 >> v1:172.21.15.117:6789/0 conn(0x7fdcf802b100 legacy=0x7fdcf80cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.514 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80c95d0 con 0x7fdcf8006a70 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcf4001690 con 0x7fdcf8006a70 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:30.516 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.534+0000 7fdcff7fe640 10 monclient: dump: 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.517 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.536+0000 7fdd06ef4640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.536+0000 7fdd06ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.536+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcf40018f0 con 0x7fdcf8006a70 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdcff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdd06ef4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdd06ef4640 4 set_mon_vals no callback set 2024-03-29T11:24:30.518 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdcff7fe640 1 -- 172.21.15.117:0/3794908956 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcf4001d00 con 0x7fdcf8006a70 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdcff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdcff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdcff7fe640 10 monclient: dump: 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdd06ef4640 10 monclient: shutdown 2024-03-29T11:24:30.519 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdd06ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.520 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.537+0000 7fdd06ef4640 1 -- 172.21.15.117:0/3794908956 >> v1:172.21.15.117:6790/0 conn(0x7fdcf8006a70 legacy=0x7fdcf80c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.538+0000 7fdd06ef4640 1 -- 172.21.15.117:0/3794908956 shutdown_connections 2024-03-29T11:24:30.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.538+0000 7fdd06ef4640 1 -- 172.21.15.117:0/3794908956 >> 172.21.15.117:0/3794908956 conn(0x7fdcf80058f0 msgr2=0x7fdcf8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.521 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.541+0000 7fdd06ef4640 1 -- 172.21.15.117:0/3794908956 shutdown_connections 2024-03-29T11:24:30.522 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.541+0000 7fdd06ef4640 1 -- 172.21.15.117:0/3794908956 wait complete. 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 10 monclient: build_initial_monmap 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 10 monclient: monmap: 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:30.523 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 librados: starting msgr at 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 librados: starting objecter 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 Processor -- start 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 -- start start 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 librados: setting wanted keys 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 1 librados: calling monclient init 2024-03-29T11:24:30.524 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.542+0000 7fdd06ef4640 10 monclient: init 2024-03-29T11:24:30.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.543+0000 7fdd06ef4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:30.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.543+0000 7fdd06ef4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:30.525 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.543+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf8006a70 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf8029c30 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf80cd540 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf80c9a10 con 0x7fdcf8029c30 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf8160b90 con 0x7fdcf8006a70 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf81617c0 con 0x7fdcf80cd540 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.546+0000 7fdd06ef4640 10 monclient(hunting): authenticate will time out at 1138.854373s 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdcf8029c30 0x7fdcf80ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59599/0 (socket says 172.21.15.117:59599) 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfffff640 1 -- 172.21.15.117:0/236137133 learned_addr learned my addr 172.21.15.117:0/236137133 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 client.?.objecter ms_handle_connect 0x7fdcf8029c30 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.531 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 client.?.objecter ms_handle_connect 0x7fdcf80cd540 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621597898 0 0) 0x7fdcf80c9a10 con 0x7fdcf8029c30 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce4004eb0 con 0x7fdcf8029c30 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699045183 0 0) 0x7fdcf81617c0 con 0x7fdcf80cd540 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf80c9a10 con 0x7fdcf80cd540 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665952271 0 0) 0x7fdce4004eb0 con 0x7fdcf8029c30 2024-03-29T11:24:30.532 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf81617c0 con 0x7fdcf8029c30 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.547+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcec001890 con 0x7fdcf8029c30 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:30.533 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470141247 0 0) 0x7fdcf80c9a10 con 0x7fdcf80cd540 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce4004eb0 con 0x7fdcf80cd540 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdd0004fdd0 con 0x7fdcf80cd540 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:30.534 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303275553 0 0) 0x7fdcf81617c0 con 0x7fdcf8029c30 2024-03-29T11:24:30.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 >> v1:172.21.15.117:6790/0 conn(0x7fdcf80cd540 legacy=0x7fdcf81610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.535 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.548+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 >> v1:172.21.15.173:6789/0 conn(0x7fdcf8006a70 legacy=0x7fdcf80cb2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.552+0000 7fdcfd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.552+0000 7fdcfd7fa640 1 monclient: found mon.a 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.552+0000 7fdcfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.552+0000 7fdcfd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.553+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf8031d50 con 0x7fdcf8029c30 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.553+0000 7fdcfd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:30.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.553+0000 7fdcfd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcec001a60 con 0x7fdcf8029c30 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdcec003870 con 0x7fdcf8029c30 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.554+0000 7fdcfd7fa640 10 monclient: dump: 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:30.538 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdcfe7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 10 monclient: _renew_subs 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf8032000 con 0x7fdcf8029c30 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 10 monclient: _renew_subs 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.555+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf80324e0 con 0x7fdcf8029c30 2024-03-29T11:24:30.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.556+0000 7fdd06ef4640 1 librados: init done 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.557+0000 7fdd06ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.557+0000 7fdd06ef4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.557+0000 7fdd06ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.557+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf81609f0 con 0x7fdcf8029c30 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.558+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fdcec0039d0 con 0x7fdcf8029c30 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.559+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fdcec084a60 con 0x7fdcf8029c30 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.559+0000 7fdd04c90640 10 monclient: get_auth_request con 0x7fdce406dd40 auth_method 0 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.560+0000 7fdcfd7fa640 10 client.4401.objecter ms_dispatch 0x7fdd00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.560+0000 7fdcfd7fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.560+0000 7fdcfd7fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:30.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.560+0000 7fdcfd7fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:24:30.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.560+0000 7fdcfd7fa640 10 client.4401.objecter ms_handle_connect 0x7fdce406dd40 2024-03-29T11:24:30.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.566+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcf81609f0 con 0x7fdcf8029c30 2024-03-29T11:24:30.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.566+0000 7fdcfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:30.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.567+0000 7fdcfd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:30.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.903+0000 7fdd06ef4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.884 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.903+0000 7fdd06ef4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.903+0000 7fdd06ef4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.903+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcf81609f0 con 0x7fdcf8029c30 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.904+0000 7fdcfd7fa640 1 -- 172.21.15.117:0/236137133 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fdcf81609f0 con 0x7fdcf8029c30 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.904+0000 7fdcfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.904+0000 7fdcfd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:30.885 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:30.886 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 10 librados: watch_flush enter 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 10 librados: watch_flush exit 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 >> v1:172.21.15.117:6816/4276038175 conn(0x7fdce406dd40 legacy=0x7fdce4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 10 monclient: shutdown 2024-03-29T11:24:30.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 >> v1:172.21.15.117:6789/0 conn(0x7fdcf8029c30 legacy=0x7fdcf80ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.908+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 shutdown_connections 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.909+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 >> 172.21.15.117:0/236137133 conn(0x7fdcf80058f0 msgr2=0x7fdcf8032d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.909+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 shutdown_connections 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.909+0000 7fdd06ef4640 1 -- 172.21.15.117:0/236137133 wait complete. 2024-03-29T11:24:30.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:30.909+0000 7fdd06ef4640 1 librados: shutdown 2024-03-29T11:24:30.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:37.714 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.913+0000 7f21df78d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.913+0000 7f21df78d640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.913+0000 7f21df78d640 10 monclient: monmap: 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:37.914311+0000 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:37.914311+0000 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.895 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.896 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:37.896 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:37.896 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:37.896 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.914+0000 7f21df78d640 1 Processor -- start 2024-03-29T11:24:37.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.914+0000 7f21df78d640 1 -- start start 2024-03-29T11:24:37.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.914+0000 7f21df78d640 10 monclient: init 2024-03-29T11:24:37.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:37.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-a con 0x7f21d8149c50 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-b con 0x7f21d80a9cc0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-c con 0x7f21d8143d00 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80ab9c0 con 0x7f21d8149c50 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80abc10 con 0x7f21d8143d00 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80aad20 con 0x7f21d80a9cc0 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21df78d640 10 monclient(hunting): authenticate will time out at 1146.224659s 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21ddd2a640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f21d8143d00 0x7f21d8151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:38808/0 (socket says 172.21.15.117:38808) 2024-03-29T11:24:37.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.915+0000 7f21ddd2a640 1 -- 172.21.15.117:0/2164087996 learned_addr learned my addr 172.21.15.117:0/2164087996 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:37.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557963636 0 0) 0x7f21d80abc10 con 0x7f21d8143d00 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c4003520 con 0x7f21d8143d00 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206877747 0 0) 0x7f21d80aad20 con 0x7f21d80a9cc0 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d80abc10 con 0x7f21d80a9cc0 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31322577 0 0) 0x7f21d80ab9c0 con 0x7f21d8149c50 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d80aad20 con 0x7f21d8149c50 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 572287393 0 0) 0x7f21c4003520 con 0x7f21d8143d00 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 >> v1:172.21.15.117:6790/0 conn(0x7f21d80a9cc0 legacy=0x7f21d81435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 >> v1:172.21.15.117:6789/0 conn(0x7f21d8149c50 legacy=0x7f21d80ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 monclient: found mon.noname-c 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80aa980 con 0x7f21d8143d00 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21c8001690 con 0x7f21d8143d00 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21cffff640 10 monclient: dump: 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21df78d640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.916+0000 7f21df78d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21c80017f0 con 0x7f21d8143d00 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 1 -- 172.21.15.117:0/2164087996 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21c8001c00 con 0x7f21d8143d00 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21cffff640 10 monclient: dump: 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.904 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21df78d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.917+0000 7f21df78d640 4 set_mon_vals no callback set 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.918+0000 7f21df78d640 10 monclient: shutdown 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.918+0000 7f21df78d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.918+0000 7f21df78d640 1 -- 172.21.15.117:0/2164087996 >> v1:172.21.15.173:6789/0 conn(0x7f21d8143d00 legacy=0x7f21d8151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.918+0000 7f21df78d640 1 -- 172.21.15.117:0/2164087996 shutdown_connections 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.918+0000 7f21df78d640 1 -- 172.21.15.117:0/2164087996 >> 172.21.15.117:0/2164087996 conn(0x7f21d80b3c50 msgr2=0x7f21d80b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.919+0000 7f21df78d640 1 -- 172.21.15.117:0/2164087996 shutdown_connections 2024-03-29T11:24:37.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.919+0000 7f21df78d640 1 -- 172.21.15.117:0/2164087996 wait complete. 2024-03-29T11:24:37.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 10 monclient: build_initial_monmap 2024-03-29T11:24:37.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 10 monclient: monmap: 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 librados: starting msgr at 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 librados: starting objecter 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 Processor -- start 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 -- start start 2024-03-29T11:24:37.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 librados: setting wanted keys 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 1 librados: calling monclient init 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.920+0000 7f21df78d640 10 monclient: init 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.921+0000 7f21df78d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.921+0000 7f21df78d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.921+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-a con 0x7f21d80a9cc0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-b con 0x7f21d8143d00 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21dd529640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f21d80a9cc0 0x7f21d8145360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58208/0 (socket says 172.21.15.117:58208) 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21dcd28640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f21d8143d00 0x7f21d813b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:39048/0 (socket says 172.21.15.117:39048) 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 10 monclient(hunting): picked mon.noname-c con 0x7f21d8149c50 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d8147a40 con 0x7f21d80a9cc0 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21dd529640 1 -- 172.21.15.117:0/2452220054 learned_addr learned my addr 172.21.15.117:0/2452220054 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d8144c20 con 0x7f21d8143d00 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d8144260 con 0x7f21d8149c50 2024-03-29T11:24:37.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21df78d640 10 monclient(hunting): authenticate will time out at 1146.231413s 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter ms_handle_connect 0x7f21d8143d00 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter ms_handle_connect 0x7f21d80a9cc0 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter ms_handle_connect 0x7f21d8149c50 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.922+0000 7f21cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797559353 0 0) 0x7f21d8147a40 con 0x7f21d80a9cc0 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21bc004eb0 con 0x7f21d80a9cc0 2024-03-29T11:24:37.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355264146 0 0) 0x7f21d8144c20 con 0x7f21d8143d00 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d8147a40 con 0x7f21d8143d00 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833417226 0 0) 0x7f21d8144260 con 0x7f21d8149c50 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d8144c20 con 0x7f21d8149c50 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455631772 0 0) 0x7f21d8147a40 con 0x7f21d8143d00 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d8144260 con 0x7f21d8143d00 2024-03-29T11:24:37.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21d40018b0 con 0x7f21d8143d00 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): dump: 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.923+0000 7f21cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:37.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3434447581 0 0) 0x7f21d8144c20 con 0x7f21d8149c50 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d8147a40 con 0x7f21d8149c50 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21c8001b00 con 0x7f21d8149c50 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): dump: 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.916 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274796799 0 0) 0x7f21bc004eb0 con 0x7f21d80a9cc0 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d8144c20 con 0x7f21d80a9cc0 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21d0001cd0 con 0x7f21d80a9cc0 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.924+0000 7f21cdffb640 10 monclient(hunting): dump: 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.917 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604889420 0 0) 0x7f21d8144260 con 0x7f21d8143d00 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 >> v1:172.21.15.117:6790/0 conn(0x7f21d8149c50 legacy=0x7f21d81e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 >> v1:172.21.15.117:6789/0 conn(0x7f21d80a9cc0 legacy=0x7f21d8145360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 1 monclient: found mon.b 2024-03-29T11:24:37.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:37.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:37.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d81dfa60 con 0x7f21d8143d00 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 10 monclient: _renew_subs 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d8146010 con 0x7f21d8143d00 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:37.921 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 10 monclient: _renew_subs 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d81df200 con 0x7f21d8143d00 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.925+0000 7f21df78d640 1 librados: init done 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21d4003610 con 0x7f21d8143d00 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f21d4003a60 con 0x7f21d8143d00 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cf7fe640 4 set_mon_vals no callback set 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.926+0000 7f21cdffb640 10 monclient: dump: 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.922 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.927+0000 7f21df78d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f21d4003bc0 con 0x7f21d8143d00 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f21d40848e0 con 0x7f21d8143d00 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 10 client.4420.objecter ms_dispatch 0x7f21d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21cdffb640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21df78d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.923 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21df78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:37.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.928+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8005080 con 0x7f21d8143d00 2024-03-29T11:24:37.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.929+0000 7f21dd529640 10 monclient: get_auth_request con 0x7f21bc06e160 auth_method 0 2024-03-29T11:24:37.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.929+0000 7f21cdffb640 10 client.4420.objecter ms_handle_connect 0x7f21bc06e160 2024-03-29T11:24:37.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.936+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21d40501c0 con 0x7f21d8143d00 2024-03-29T11:24:37.924 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.936+0000 7f21cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:37.925 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:37.936+0000 7f21cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:38.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.298+0000 7f21df78d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.298+0000 7f21df78d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.298+0000 7f21df78d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:24:38.280 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.298+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21b8005330 con 0x7f21d8143d00 2024-03-29T11:24:38.281 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:38.281 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:38.281 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.299+0000 7f21cdffb640 1 -- 172.21.15.117:0/2452220054 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f21d40520f0 con 0x7f21d8143d00 2024-03-29T11:24:38.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.299+0000 7f21cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:38.282 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.299+0000 7f21cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:38.284 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.302+0000 7f21df78d640 10 librados: watch_flush enter 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.302+0000 7f21df78d640 10 librados: watch_flush exit 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.302+0000 7f21df78d640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.302+0000 7f21df78d640 10 client.4420.objecter successfully canceled tick 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 >> v1:172.21.15.117:6816/4276038175 conn(0x7f21bc06e160 legacy=0x7f21bc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 10 monclient: shutdown 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:38.285 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 >> v1:172.21.15.173:6789/0 conn(0x7f21d8143d00 legacy=0x7f21d813b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:38.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 shutdown_connections 2024-03-29T11:24:38.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 >> 172.21.15.117:0/2452220054 conn(0x7f21d80b3c50 msgr2=0x7f21d8141f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:38.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 shutdown_connections 2024-03-29T11:24:38.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 -- 172.21.15.117:0/2452220054 wait complete. 2024-03-29T11:24:38.286 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:38.303+0000 7f21df78d640 1 librados: shutdown 2024-03-29T11:24:38.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:45.108 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.264+0000 7f862dc1e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.264+0000 7f862dc1e640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.264+0000 7f862dc1e640 10 monclient: monmap: 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:45.265112+0000 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:45.265112+0000 2024-03-29T11:24:45.247 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.248 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.248 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:45.248 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:45.248 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:45.248 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.265+0000 7f862dc1e640 1 Processor -- start 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.265+0000 7f862dc1e640 1 -- start start 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.265+0000 7f862dc1e640 10 monclient: init 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-c con 0x7f86280b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-b con 0x7f862814a700 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-a con 0x7f86280ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280aba70 con 0x7f86280ac5a0 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280aecb0 con 0x7f86280b0770 2024-03-29T11:24:45.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280af150 con 0x7f862814a700 2024-03-29T11:24:45.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.266+0000 7f862dc1e640 10 monclient(hunting): authenticate will time out at 1153.575767s 2024-03-29T11:24:45.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.268+0000 7f861ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f862814a700 0x7f86280abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:41522/0 (socket says 172.21.15.117:41522) 2024-03-29T11:24:45.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.268+0000 7f861ffff640 1 -- 172.21.15.117:0/1223358354 learned_addr learned my addr 172.21.15.117:0/1223358354 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:45.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142630650 0 0) 0x7f86280af150 con 0x7f862814a700 2024-03-29T11:24:45.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8614003520 con 0x7f862814a700 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574877824 0 0) 0x7f8614003520 con 0x7f862814a700 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 >> v1:172.21.15.173:6789/0 conn(0x7f86280b0770 legacy=0x7f8628149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 >> v1:172.21.15.117:6789/0 conn(0x7f86280ac5a0 legacy=0x7f8628151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:45.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 monclient: found mon.noname-b 2024-03-29T11:24:45.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280ac9d0 con 0x7f862814a700 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8610001690 con 0x7f862814a700 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f8626ffd640 10 monclient: dump: 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:45.256 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f862dc1e640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.269+0000 7f862dc1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86100018f0 con 0x7f862814a700 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 1 -- 172.21.15.117:0/1223358354 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8610001d00 con 0x7f862814a700 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f8626ffd640 10 monclient: dump: 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.257 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f862dc1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.270+0000 7f862dc1e640 4 set_mon_vals no callback set 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 10 monclient: shutdown 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 1 -- 172.21.15.117:0/1223358354 >> v1:172.21.15.117:6790/0 conn(0x7f862814a700 legacy=0x7f86280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 1 -- 172.21.15.117:0/1223358354 shutdown_connections 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 1 -- 172.21.15.117:0/1223358354 >> 172.21.15.117:0/1223358354 conn(0x7f862806b660 msgr2=0x7f862806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 1 -- 172.21.15.117:0/1223358354 shutdown_connections 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.271+0000 7f862dc1e640 1 -- 172.21.15.117:0/1223358354 wait complete. 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 10 monclient: build_initial_monmap 2024-03-29T11:24:45.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 10 monclient: monmap: 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 librados: starting msgr at 2024-03-29T11:24:45.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 librados: starting objecter 2024-03-29T11:24:45.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 Processor -- start 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 -- start start 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 librados: setting wanted keys 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 1 librados: calling monclient init 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.273+0000 7f862dc1e640 10 monclient: init 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.274+0000 7f862dc1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.274+0000 7f862dc1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.274+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-a con 0x7f86280ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-c con 0x7f86280b0770 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 10 monclient(hunting): picked mon.noname-b con 0x7f862814a700 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:45.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862812a4b0 con 0x7f86280ac5a0 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86281281f0 con 0x7f862814a700 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8628127dc0 con 0x7f86280b0770 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f862dc1e640 10 monclient(hunting): authenticate will time out at 1153.584173s 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f861ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f86280b0770 0x7f8628128680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:54654/0 (socket says 172.21.15.117:54654) 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f861ffff640 1 -- 172.21.15.117:0/931380019 learned_addr learned my addr 172.21.15.117:0/931380019 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f8624ff9640 10 client.?.objecter ms_handle_connect 0x7f86280b0770 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.275+0000 7f8624ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087726525 0 0) 0x7f8628127dc0 con 0x7f86280b0770 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8608004eb0 con 0x7f86280b0770 2024-03-29T11:24:45.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587431622 0 0) 0x7f8608004eb0 con 0x7f86280b0770 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8628127dc0 con 0x7f86280b0770 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 client.?.objecter ms_handle_connect 0x7f862814a700 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8610001650 con 0x7f86280b0770 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.265 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.276+0000 7f8624ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1686675917 0 0) 0x7f8628127dc0 con 0x7f86280b0770 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 >> v1:172.21.15.173:6789/0 conn(0x7f862814a700 legacy=0x7f8628128d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 >> v1:172.21.15.117:6789/0 conn(0x7f86280ac5a0 legacy=0x7f8628146f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 1 monclient: found mon.c 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628147bd0 con 0x7f86280b0770 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f8624ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 10 monclient: _renew_subs 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86281df600 con 0x7f86280b0770 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 10 monclient: _renew_subs 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8628147840 con 0x7f86280b0770 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.277+0000 7f862dc1e640 1 librados: init done 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.279+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8610004a90 con 0x7f86280b0770 2024-03-29T11:24:45.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.279+0000 7f8624ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.279+0000 7f86267fc640 4 set_mon_vals no callback set 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8610004e60 con 0x7f86280b0770 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f8624ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f8624ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f8624ff9640 10 monclient: dump: 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:45.268 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f862dc1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f862dc1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f862dc1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.280+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c005080 con 0x7f86280b0770 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.281+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f8610004fc0 con 0x7f86280b0770 2024-03-29T11:24:45.269 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f8610084cc0 con 0x7f86280b0770 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f8624ff9640 10 client.4484.objecter ms_dispatch 0x7f8628000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f8624ff9640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f8624ff9640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f8624ff9640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.282+0000 7f86277fe640 10 monclient: get_auth_request con 0x7f860806dbd0 auth_method 0 2024-03-29T11:24:45.270 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.283+0000 7f8624ff9640 10 client.4484.objecter ms_handle_connect 0x7f860806dbd0 2024-03-29T11:24:45.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.289+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86100505a0 con 0x7f86280b0770 2024-03-29T11:24:45.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.289+0000 7f8624ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:45.271 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.289+0000 7f8624ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:45.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.604+0000 7f862dc1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.604+0000 7f862dc1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.604+0000 7f862dc1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.604+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f860c005330 con 0x7f86280b0770 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.605+0000 7f8624ff9640 1 -- 172.21.15.117:0/931380019 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f86100524d0 con 0x7f86280b0770 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.605+0000 7f8624ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:45.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.605+0000 7f8624ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:45.588 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:45.588 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:45.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 10 librados: watch_flush enter 2024-03-29T11:24:45.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 10 librados: watch_flush exit 2024-03-29T11:24:45.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 >> v1:172.21.15.117:6816/4276038175 conn(0x7f860806dbd0 legacy=0x7f8608070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 10 monclient: shutdown 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.609+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 >> v1:172.21.15.117:6790/0 conn(0x7f86280b0770 legacy=0x7f8628128680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:45.592 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.610+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 shutdown_connections 2024-03-29T11:24:45.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.610+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 >> 172.21.15.117:0/931380019 conn(0x7f862806b660 msgr2=0x7f8628071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:45.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.610+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 shutdown_connections 2024-03-29T11:24:45.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.610+0000 7f862dc1e640 1 -- 172.21.15.117:0/931380019 wait complete. 2024-03-29T11:24:45.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:45.610+0000 7f862dc1e640 1 librados: shutdown 2024-03-29T11:24:45.608 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:52.415 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:52.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.575+0000 7fb76253d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:52.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.575+0000 7fb76253d640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.575+0000 7fb76253d640 10 monclient: monmap: 2024-03-29T11:24:52.558 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:52.576641+0000 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:52.576641+0000 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.577+0000 7fb76253d640 1 Processor -- start 2024-03-29T11:24:52.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.577+0000 7fb76253d640 1 -- start start 2024-03-29T11:24:52.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.577+0000 7fb76253d640 10 monclient: init 2024-03-29T11:24:52.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.577+0000 7fb76253d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-a con 0x7fb75c06e380 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-c con 0x7fb75c06e760 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-b con 0x7fb75c13b030 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c13e4e0 con 0x7fb75c06e380 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c070010 con 0x7fb75c06e760 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c0702f0 con 0x7fb75c13b030 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb76253d640 10 monclient(hunting): authenticate will time out at 1160.887127s 2024-03-29T11:24:52.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb760ada640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb75c13b030 0x7fb75c13b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45338/0 (socket says 172.21.15.117:45338) 2024-03-29T11:24:52.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.578+0000 7fb760ada640 1 -- 172.21.15.117:0/3755438941 learned_addr learned my addr 172.21.15.117:0/3755438941 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:52.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.580+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663808142 0 0) 0x7fb75c0702f0 con 0x7fb75c13b030 2024-03-29T11:24:52.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb744003520 con 0x7fb75c13b030 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046796789 0 0) 0x7fb750001110 con 0x7fb75c06e380 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75c0702f0 con 0x7fb75c06e380 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1962599565 0 0) 0x7fb744003520 con 0x7fb75c13b030 2024-03-29T11:24:52.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.581+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 >> v1:172.21.15.173:6789/0 conn(0x7fb75c06e760 legacy=0x7fb75c13e960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.584+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 >> v1:172.21.15.117:6789/0 conn(0x7fb75c06e380 legacy=0x7fb75c0ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.584+0000 7fb75affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 1 monclient: found mon.noname-b 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c0705d0 con 0x7fb75c13b030 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb754001a10 con 0x7fb75c13b030 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb76253d640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:24:52.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb76253d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: dump: 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:52.571 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb76253d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb754001b70 con 0x7fb75c13b030 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb76253d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb76253d640 4 set_mon_vals no callback set 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 1 -- 172.21.15.117:0/3755438941 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb754001fc0 con 0x7fb75c13b030 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.585+0000 7fb75affd640 10 monclient: dump: 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.572 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.586+0000 7fb76253d640 10 monclient: shutdown 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.586+0000 7fb76253d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.586+0000 7fb76253d640 1 -- 172.21.15.117:0/3755438941 >> v1:172.21.15.117:6790/0 conn(0x7fb75c13b030 legacy=0x7fb75c13b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.587+0000 7fb76253d640 1 -- 172.21.15.117:0/3755438941 shutdown_connections 2024-03-29T11:24:52.574 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.587+0000 7fb76253d640 1 -- 172.21.15.117:0/3755438941 >> 172.21.15.117:0/3755438941 conn(0x7fb75c067570 msgr2=0x7fb75c0b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.588+0000 7fb76253d640 1 -- 172.21.15.117:0/3755438941 shutdown_connections 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.588+0000 7fb76253d640 1 -- 172.21.15.117:0/3755438941 wait complete. 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.589+0000 7fb76253d640 10 monclient: build_initial_monmap 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.589+0000 7fb76253d640 10 monclient: monmap: 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:52.575 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.589+0000 7fb76253d640 1 librados: starting msgr at 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.589+0000 7fb76253d640 1 librados: starting objecter 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.589+0000 7fb76253d640 1 Processor -- start 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.590+0000 7fb76253d640 1 -- start start 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.590+0000 7fb76253d640 1 librados: setting wanted keys 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.590+0000 7fb76253d640 1 librados: calling monclient init 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.590+0000 7fb76253d640 10 monclient: init 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-a con 0x7fb75c06e380 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-b con 0x7fb75c06e760 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:52.576 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient(hunting): picked mon.noname-c con 0x7fb75c13b030 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:52.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c1e4be0 con 0x7fb75c06e380 2024-03-29T11:24:52.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c1e1870 con 0x7fb75c06e760 2024-03-29T11:24:52.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c136430 con 0x7fb75c13b030 2024-03-29T11:24:52.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:52.577 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.591+0000 7fb76253d640 10 monclient(hunting): authenticate will time out at 1160.900787s 2024-03-29T11:24:52.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb760ada640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb75c13b030 0x7fb75c1e1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40342/0 (socket says 172.21.15.117:40342) 2024-03-29T11:24:52.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb760ada640 1 -- 172.21.15.117:0/3096727592 learned_addr learned my addr 172.21.15.117:0/3096727592 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:52.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 client.?.objecter ms_handle_connect 0x7fb75c13b030 2024-03-29T11:24:52.578 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002165692 0 0) 0x7fb75c136430 con 0x7fb75c13b030 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748004eb0 con 0x7fb75c13b030 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 client.?.objecter ms_handle_connect 0x7fb75c06e380 2024-03-29T11:24:52.579 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.594+0000 7fb758ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:52.580 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3639355622 0 0) 0x7fb748004eb0 con 0x7fb75c13b030 2024-03-29T11:24:52.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb75c136430 con 0x7fb75c13b030 2024-03-29T11:24:52.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb754001e80 con 0x7fb75c13b030 2024-03-29T11:24:52.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:52.583 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.597+0000 7fb758ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:52.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769957162 0 0) 0x7fb750001070 con 0x7fb75c06e380 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748004eb0 con 0x7fb75c06e380 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3926592225 0 0) 0x7fb75c136430 con 0x7fb75c13b030 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 >> v1:172.21.15.173:6789/0 conn(0x7fb75c06e760 legacy=0x7fb75c13ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 >> v1:172.21.15.117:6789/0 conn(0x7fb75c06e380 legacy=0x7fb75c136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 monclient: found mon.c 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c1369f0 con 0x7fb75c13b030 2024-03-29T11:24:52.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.598+0000 7fb758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 10 monclient: _renew_subs 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c1351d0 con 0x7fb75c13b030 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 10 monclient: _renew_subs 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c0b3680 con 0x7fb75c13b030 2024-03-29T11:24:52.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.599+0000 7fb76253d640 1 librados: init done 2024-03-29T11:24:52.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb76253d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb76253d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb76253d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c0b3980 con 0x7fb75c13b030 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb754001fe0 con 0x7fb75c13b030 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb754005110 con 0x7fb75c13b030 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 10 monclient: dump: 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:52.589 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fb75401b120 con 0x7fb75c13b030 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.601+0000 7fb75a7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fb754085100 con 0x7fb75c13b030 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb758ff9640 10 client.4490.objecter ms_dispatch 0x7fb75c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb758ff9640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb758ff9640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb758ff9640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.602+0000 7fb75bfff640 10 monclient: get_auth_request con 0x7fb74806dda0 auth_method 0 2024-03-29T11:24:52.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.603+0000 7fb758ff9640 10 client.4490.objecter ms_handle_connect 0x7fb74806dda0 2024-03-29T11:24:52.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.610+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb754050a60 con 0x7fb75c13b030 2024-03-29T11:24:52.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.610+0000 7fb758ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:52.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.610+0000 7fb758ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:52.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.925+0000 7fb76253d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.925+0000 7fb76253d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.925+0000 7fb76253d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:52.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.925+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb75c136b80 con 0x7fb75c13b030 2024-03-29T11:24:52.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.926+0000 7fb758ff9640 1 -- 172.21.15.117:0/3096727592 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fb754052990 con 0x7fb75c13b030 2024-03-29T11:24:52.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.926+0000 7fb758ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:52.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.926+0000 7fb758ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:52.910 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:24:52.910 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:52.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 10 librados: watch_flush enter 2024-03-29T11:24:52.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 10 librados: watch_flush exit 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 >> v1:172.21.15.117:6816/4276038175 conn(0x7fb74806dda0 legacy=0x7fb748070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 10 monclient: shutdown 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.929+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 >> v1:172.21.15.117:6790/0 conn(0x7fb75c13b030 legacy=0x7fb75c1e1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.930+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 shutdown_connections 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.930+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 >> 172.21.15.117:0/3096727592 conn(0x7fb75c067570 msgr2=0x7fb75c06eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.930+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 shutdown_connections 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.930+0000 7fb76253d640 1 -- 172.21.15.117:0/3096727592 wait complete. 2024-03-29T11:24:52.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:52.930+0000 7fb76253d640 1 librados: shutdown 2024-03-29T11:24:52.925 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:59.733 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:59.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.900+0000 7feeeadba640 10 monclient: get_monmap_and_config 2024-03-29T11:24:59.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.900+0000 7feeeadba640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.883 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.900+0000 7feeeadba640 10 monclient: monmap: 2024-03-29T11:24:59.883 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:59.883 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:24:59.901585+0000 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:24:59.901585+0000 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:24:59.884 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.902+0000 7feeeadba640 1 Processor -- start 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.902+0000 7feeeadba640 1 -- start start 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.902+0000 7feeeadba640 10 monclient: init 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-c con 0x7feee41422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-a con 0x7feee4142690 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:59.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-b con 0x7feee40b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee4142c80 con 0x7feee4142690 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee40aae40 con 0x7feee41422b0 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee413fbd0 con 0x7feee40b17d0 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.903+0000 7feeeadba640 10 monclient(hunting): authenticate will time out at 1168.212167s 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.904+0000 7feee9357640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7feee40b17d0 0x7feee40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48330/0 (socket says 172.21.15.117:48330) 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.904+0000 7feee9357640 1 -- 172.21.15.117:0/1612905459 learned_addr learned my addr 172.21.15.117:0/1612905459 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258324119 0 0) 0x7feee413fbd0 con 0x7feee40b17d0 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:24:59.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc003520 con 0x7feee40b17d0 2024-03-29T11:24:59.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 387808695 0 0) 0x7feecc003520 con 0x7feee40b17d0 2024-03-29T11:24:59.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.905+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 >> v1:172.21.15.173:6789/0 conn(0x7feee41422b0 legacy=0x7feee413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.907+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 >> v1:172.21.15.117:6789/0 conn(0x7feee4142690 legacy=0x7feee40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4151940 con 0x7feee40b17d0 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7feed4001690 con 0x7feee40b17d0 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.908+0000 7feee37fe640 10 monclient: dump: 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feeeadba640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feeeadba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feed40018f0 con 0x7feee40b17d0 2024-03-29T11:24:59.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feee37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feeeadba640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.909+0000 7feeeadba640 4 set_mon_vals no callback set 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feee37fe640 1 -- 172.21.15.117:0/1612905459 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7feed4001cc0 con 0x7feee40b17d0 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feee37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feee37fe640 10 monclient: dump: 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:59.895 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feeeadba640 10 monclient: shutdown 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feeeadba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feeeadba640 1 -- 172.21.15.117:0/1612905459 >> v1:172.21.15.117:6790/0 conn(0x7feee40b17d0 legacy=0x7feee40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feeeadba640 1 -- 172.21.15.117:0/1612905459 shutdown_connections 2024-03-29T11:24:59.896 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.910+0000 7feeeadba640 1 -- 172.21.15.117:0/1612905459 >> 172.21.15.117:0/1612905459 conn(0x7feee4066a00 msgr2=0x7feee4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:59.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.912+0000 7feeeadba640 1 -- 172.21.15.117:0/1612905459 shutdown_connections 2024-03-29T11:24:59.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.914+0000 7feeeadba640 1 -- 172.21.15.117:0/1612905459 wait complete. 2024-03-29T11:24:59.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.915+0000 7feeeadba640 10 monclient: build_initial_monmap 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.915+0000 7feeeadba640 10 monclient: monmap: 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:24:59.901 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.915+0000 7feeeadba640 1 librados: starting msgr at 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.915+0000 7feeeadba640 1 librados: starting objecter 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.915+0000 7feeeadba640 1 Processor -- start 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.916+0000 7feeeadba640 1 -- start start 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.916+0000 7feeeadba640 1 librados: setting wanted keys 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.916+0000 7feeeadba640 1 librados: calling monclient init 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.916+0000 7feeeadba640 10 monclient: init 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-b con 0x7feee40b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-a con 0x7feee41422b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient(hunting): picked mon.noname-c con 0x7feee4142690 addr v1:172.21.15.117:6790/0 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee406ef30 con 0x7feee41422b0 2024-03-29T11:24:59.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee406f210 con 0x7feee40b17d0 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee41318c0 con 0x7feee4142690 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feeeadba640 10 monclient(hunting): authenticate will time out at 1168.226150s 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee9357640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7feee4142690 0x7feee4070970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53576/0 (socket says 172.21.15.117:53576) 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee9357640 1 -- 172.21.15.117:0/3090977252 learned_addr learned my addr 172.21.15.117:0/3090977252 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 client.?.objecter ms_handle_connect 0x7feee4142690 2024-03-29T11:24:59.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610797735 0 0) 0x7feee41318c0 con 0x7feee4142690 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:24:59.905 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.917+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed0004eb0 con 0x7feee4142690 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842049515 0 0) 0x7feed0004eb0 con 0x7feee4142690 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feed0005300 con 0x7feee4142690 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7feed4001c00 con 0x7feee4142690 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 10 monclient(hunting): dump: 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:59.906 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 181909733 0 0) 0x7feed0005300 con 0x7feee4142690 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.918+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 >> v1:172.21.15.173:6789/0 conn(0x7feee40b17d0 legacy=0x7feee413b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 >> v1:172.21.15.117:6789/0 conn(0x7feee41422b0 legacy=0x7feee4132110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 1 monclient: found mon.c 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4132820 con 0x7feee4142690 2024-03-29T11:24:59.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:59.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:59.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:59.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 client.?.objecter ms_handle_connect 0x7feee41422b0 2024-03-29T11:24:59.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feee17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feeeadba640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feeeadba640 10 monclient: _renew_subs 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feeeadba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.920+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee41debe0 con 0x7feee4142690 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feed4004bf0 con 0x7feee4142690 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7feed4001c00 con 0x7feee4142690 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:59.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee17fa640 10 monclient: dump: 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:24:59.915 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feeeadba640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feeeadba640 10 monclient: _renew_subs 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feeeadba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee406e070 con 0x7feee4142690 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feeeadba640 1 librados: init done 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.923+0000 7feee2ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.925+0000 7feeeadba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.925+0000 7feeeadba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.925+0000 7feeeadba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.925+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4131a50 con 0x7feee4142690 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.926+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7feed401ac50 con 0x7feee4142690 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7feed4084660 con 0x7feee4142690 2024-03-29T11:24:59.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee17fa640 10 client.4496.objecter ms_dispatch 0x7feee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee17fa640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee17fa640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee17fa640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.927+0000 7feee8b56640 10 monclient: get_auth_request con 0x7feed006e1e0 auth_method 0 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.928+0000 7feee17fa640 10 client.4496.objecter ms_handle_connect 0x7feed006e1e0 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.932+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feed404ff40 con 0x7feee4142690 2024-03-29T11:24:59.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.932+0000 7feee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:59.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:24:59.932+0000 7feee17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.245+0000 7feeeadba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.245+0000 7feeeadba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.245+0000 7feeeadba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:00.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.245+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee4131a50 con 0x7feee4142690 2024-03-29T11:25:00.231 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:00.231 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:00.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.246+0000 7feee17fa640 1 -- 172.21.15.117:0/3090977252 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7feee4131a50 con 0x7feee4142690 2024-03-29T11:25:00.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.246+0000 7feee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:00.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.246+0000 7feee17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:00.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 10 librados: watch_flush enter 2024-03-29T11:25:00.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 10 librados: watch_flush exit 2024-03-29T11:25:00.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:25:00.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 >> v1:172.21.15.117:6816/4276038175 conn(0x7feed006e1e0 legacy=0x7feed00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 10 monclient: shutdown 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.250+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 >> v1:172.21.15.117:6790/0 conn(0x7feee4142690 legacy=0x7feee4070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.251+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 shutdown_connections 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.251+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 >> 172.21.15.117:0/3090977252 conn(0x7feee4066a00 msgr2=0x7feee40b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.251+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 shutdown_connections 2024-03-29T11:25:00.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.252+0000 7feeeadba640 1 -- 172.21.15.117:0/3090977252 wait complete. 2024-03-29T11:25:00.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:00.252+0000 7feeeadba640 1 librados: shutdown 2024-03-29T11:25:00.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:07.065 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:07.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.218+0000 7f831f2c0640 10 monclient: get_monmap_and_config 2024-03-29T11:25:07.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.218+0000 7f831f2c0640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.201 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.218+0000 7f831f2c0640 10 monclient: monmap: 2024-03-29T11:25:07.201 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:07.219288+0000 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:07.219288+0000 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:07.202 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.220+0000 7f831f2c0640 1 Processor -- start 2024-03-29T11:25:07.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.220+0000 7f831f2c0640 1 -- start start 2024-03-29T11:25:07.203 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.220+0000 7f831f2c0640 10 monclient: init 2024-03-29T11:25:07.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:07.204 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f83181457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f831813f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f83180a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180aaef0 con 0x7f83181457f0 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318145bd0 con 0x7f83180a9d70 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318145eb0 con 0x7f831813f650 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.221+0000 7f831f2c0640 10 monclient(hunting): authenticate will time out at 1175.530937s 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.222+0000 7f831d85d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f83180a9d70 0x7f831814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:47764/0 (socket says 172.21.15.117:47764) 2024-03-29T11:25:07.205 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.222+0000 7f831d85d640 1 -- 172.21.15.117:0/864187351 learned_addr learned my addr 172.21.15.117:0/864187351 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701257724 0 0) 0x7f8318145eb0 con 0x7f831813f650 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003520 con 0x7f831813f650 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233140202 0 0) 0x7f8304003520 con 0x7f831813f650 2024-03-29T11:25:07.209 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 >> v1:172.21.15.173:6789/0 conn(0x7f83180a9d70 legacy=0x7f831814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 >> v1:172.21.15.117:6789/0 conn(0x7f83181457f0 legacy=0x7f831813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 monclient: found mon.noname-b 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318146340 con 0x7f831813f650 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.210 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83100019f0 con 0x7f831813f650 2024-03-29T11:25:07.213 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f830ffff640 10 monclient: dump: 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f831f2c0640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:25:07.214 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.224+0000 7f831f2c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8310001c50 con 0x7f831813f650 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f831f2c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 1 -- 172.21.15.117:0/864187351 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83100020a0 con 0x7f831813f650 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f831f2c0640 4 set_mon_vals no callback set 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f830ffff640 10 monclient: dump: 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.215 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f831f2c0640 10 monclient: shutdown 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f831f2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.225+0000 7f831f2c0640 1 -- 172.21.15.117:0/864187351 >> v1:172.21.15.117:6790/0 conn(0x7f831813f650 legacy=0x7f83180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.226+0000 7f831f2c0640 1 -- 172.21.15.117:0/864187351 shutdown_connections 2024-03-29T11:25:07.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.226+0000 7f831f2c0640 1 -- 172.21.15.117:0/864187351 >> 172.21.15.117:0/864187351 conn(0x7f8318066a00 msgr2=0x7f8318066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.226+0000 7f831f2c0640 1 -- 172.21.15.117:0/864187351 shutdown_connections 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.226+0000 7f831f2c0640 1 -- 172.21.15.117:0/864187351 wait complete. 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 10 monclient: build_initial_monmap 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 10 monclient: monmap: 2024-03-29T11:25:07.224 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 librados: starting msgr at 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 librados: starting objecter 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 Processor -- start 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 -- start start 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 librados: setting wanted keys 2024-03-29T11:25:07.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 1 librados: calling monclient init 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.227+0000 7f831f2c0640 10 monclient: init 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.228+0000 7f831f2c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.228+0000 7f831f2c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f83180a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f831813f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f83181457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180aee60 con 0x7f83180a9d70 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180af0d0 con 0x7f831813f650 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180aca00 con 0x7f83181457f0 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831f2c0640 10 monclient(hunting): authenticate will time out at 1175.538453s 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831d85d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f83181457f0 0x7f83180ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:54086/0 (socket says 172.21.15.117:54086) 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f831d85d640 1 -- 172.21.15.117:0/4270871191 learned_addr learned my addr 172.21.15.117:0/4270871191 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:07.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f830dffb640 10 client.?.objecter ms_handle_connect 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.229+0000 7f830dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660298945 0 0) 0x7f83180aca00 con 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003160 con 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273514680 0 0) 0x7f8304003160 con 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83180aca00 con 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.230+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8314001510 con 0x7f83181457f0 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): dump: 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:07.227 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 client.?.objecter ms_handle_connect 0x7f831813f650 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797993637 0 0) 0x7f83180af0d0 con 0x7f831813f650 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003160 con 0x7f831813f650 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 client.?.objecter ms_handle_connect 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462675745 0 0) 0x7f83180aee60 con 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.231+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83180af0d0 con 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3368013197 0 0) 0x7f83180af0d0 con 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83180aee60 con 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f83080016f0 con 0x7f83180a9d70 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:07.229 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 10 monclient(hunting): dump: 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 12910704 0 0) 0x7f83180aca00 con 0x7f83181457f0 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 >> v1:172.21.15.173:6789/0 conn(0x7f831813f650 legacy=0x7f831813a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.232+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 >> v1:172.21.15.117:6789/0 conn(0x7f83180a9d70 legacy=0x7f8318141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 1 monclient: found mon.c 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318136670 con 0x7f83181457f0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8314001ae0 con 0x7f83181457f0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8314004cc0 con 0x7f83181457f0 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830dffb640 10 monclient: dump: 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:07.231 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 10 monclient: _renew_subs 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83180acf20 con 0x7f83181457f0 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f830f7fe640 4 set_mon_vals no callback set 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 10 monclient: _renew_subs 2024-03-29T11:25:07.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83181f4290 con 0x7f83181457f0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.233+0000 7f831f2c0640 1 librados: init done 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.235+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f831401ac90 con 0x7f83181457f0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f831d05c640 10 monclient: get_auth_request con 0x7f830406e180 auth_method 0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f830dffb640 10 client.4502.objecter ms_handle_connect 0x7f830406e180 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f8314084a60 con 0x7f83181457f0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f830dffb640 10 client.4502.objecter ms_dispatch 0x7f8318000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f830dffb640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.236+0000 7f830dffb640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.237+0000 7f830dffb640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.237+0000 7f831f2c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.237+0000 7f831f2c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.237+0000 7f831f2c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.237+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831813a860 con 0x7f83181457f0 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.244+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8314050340 con 0x7f83181457f0 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.244+0000 7f830dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:07.234 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.244+0000 7f830dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:07.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.576+0000 7f831f2c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f831f2c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f831f2c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83180ad0b0 con 0x7f83181457f0 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f830dffb640 1 -- 172.21.15.117:0/4270871191 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f8314052270 con 0x7f83181457f0 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f830dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.577+0000 7f830dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:07.561 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.580+0000 7f831f2c0640 10 librados: watch_flush enter 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.580+0000 7f831f2c0640 10 librados: watch_flush exit 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.580+0000 7f831f2c0640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.580+0000 7f831f2c0640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.581+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 >> v1:172.21.15.117:6816/4276038175 conn(0x7f830406e180 legacy=0x7f8304070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.581+0000 7f831f2c0640 10 monclient: shutdown 2024-03-29T11:25:07.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.581+0000 7f831f2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.581+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 >> v1:172.21.15.117:6790/0 conn(0x7f83181457f0 legacy=0x7f83180ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.582+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 shutdown_connections 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.582+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 >> 172.21.15.117:0/4270871191 conn(0x7f8318066a00 msgr2=0x7f83180b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.582+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 shutdown_connections 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.582+0000 7f831f2c0640 1 -- 172.21.15.117:0/4270871191 wait complete. 2024-03-29T11:25:07.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:07.582+0000 7f831f2c0640 1 librados: shutdown 2024-03-29T11:25:07.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:14.390 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.552+0000 7f1f57c27640 10 monclient: get_monmap_and_config 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.552+0000 7f1f57c27640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.552+0000 7f1f57c27640 10 monclient: monmap: 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:14.553736+0000 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:14.553736+0000 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.536 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.537 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:14.537 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:14.537 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:14.537 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.537 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.554+0000 7f1f57c27640 1 Processor -- start 2024-03-29T11:25:14.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.554+0000 7f1f57c27640 1 -- start start 2024-03-29T11:25:14.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.554+0000 7f1f57c27640 10 monclient: init 2024-03-29T11:25:14.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-b con 0x7f1f501457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-c con 0x7f1f5013f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-a con 0x7f1f500a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f500aaef0 con 0x7f1f500a9d70 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f50145bd0 con 0x7f1f5013f650 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f50145eb0 con 0x7f1f501457f0 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f57c27640 10 monclient(hunting): authenticate will time out at 1182.864596s 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f559c3640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1f501457f0 0x7f1f5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46706/0 (socket says 172.21.15.117:46706) 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f559c3640 1 -- 172.21.15.117:0/1300424500 learned_addr learned my addr 172.21.15.117:0/1300424500 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.555+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31360626 0 0) 0x7f1f50145eb0 con 0x7f1f501457f0 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:25:14.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f3c003520 con 0x7f1f501457f0 2024-03-29T11:25:14.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380879524 0 0) 0x7f1f3c003520 con 0x7f1f501457f0 2024-03-29T11:25:14.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.556+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 >> v1:172.21.15.173:6789/0 conn(0x7f1f5013f650 legacy=0x7f1f500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 >> v1:172.21.15.117:6789/0 conn(0x7f1f500a9d70 legacy=0x7f1f5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.543 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 1 monclient: found mon.noname-b 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f50146340 con 0x7f1f501457f0 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1f4c001690 con 0x7f1f501457f0 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.560+0000 7f1f549c1640 10 monclient: dump: 2024-03-29T11:25:14.544 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:14.545 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f4c001860 con 0x7f1f501457f0 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 1 -- 172.21.15.117:0/1300424500 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1f4c001cb0 con 0x7f1f501457f0 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.561+0000 7f1f549c1640 10 monclient: dump: 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:14.547 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.562+0000 7f1f57c27640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.562+0000 7f1f57c27640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.562+0000 7f1f57c27640 4 set_mon_vals no callback set 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.563+0000 7f1f57c27640 10 monclient: shutdown 2024-03-29T11:25:14.548 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.563+0000 7f1f57c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.563+0000 7f1f57c27640 1 -- 172.21.15.117:0/1300424500 >> v1:172.21.15.117:6790/0 conn(0x7f1f501457f0 legacy=0x7f1f5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.563+0000 7f1f57c27640 1 -- 172.21.15.117:0/1300424500 shutdown_connections 2024-03-29T11:25:14.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.563+0000 7f1f57c27640 1 -- 172.21.15.117:0/1300424500 >> 172.21.15.117:0/1300424500 conn(0x7f1f50066a00 msgr2=0x7f1f50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.564+0000 7f1f57c27640 1 -- 172.21.15.117:0/1300424500 shutdown_connections 2024-03-29T11:25:14.549 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.564+0000 7f1f57c27640 1 -- 172.21.15.117:0/1300424500 wait complete. 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 10 monclient: build_initial_monmap 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 10 monclient: monmap: 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:14.550 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 librados: starting msgr at 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 librados: starting objecter 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 Processor -- start 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 -- start start 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 librados: setting wanted keys 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 1 librados: calling monclient init 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.566+0000 7f1f57c27640 10 monclient: init 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.567+0000 7f1f57c27640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.567+0000 7f1f57c27640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:14.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.567+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-a con 0x7f1f500a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:14.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.567+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-b con 0x7f1f5013f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.568+0000 7f1f57c27640 10 monclient(hunting): picked mon.noname-c con 0x7f1f501457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.568+0000 7f1f561c4640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f1f501457f0 0x7f1f501ee940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48706/0 (socket says 172.21.15.117:48706) 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.568+0000 7f1f561c4640 1 -- 172.21.15.117:0/1673282869 learned_addr learned my addr 172.21.15.117:0/1673282869 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.568+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f50144140 con 0x7f1f500a9d70 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.568+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f50144d50 con 0x7f1f5013f650 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f50145030 con 0x7f1f501457f0 2024-03-29T11:25:14.554 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f57c27640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f57c27640 10 monclient(hunting): authenticate will time out at 1182.878364s 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 client.?.objecter ms_handle_connect 0x7f1f501457f0 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216088963 0 0) 0x7f1f50145030 con 0x7f1f501457f0 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.569+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f34004eb0 con 0x7f1f501457f0 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 640094095 0 0) 0x7f1f34004eb0 con 0x7f1f501457f0 2024-03-29T11:25:14.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f50145030 con 0x7f1f501457f0 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1f400018b0 con 0x7f1f501457f0 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 10 monclient(hunting): dump: 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:14.556 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1762449612 0 0) 0x7f1f50145030 con 0x7f1f501457f0 2024-03-29T11:25:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.570+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 >> v1:172.21.15.173:6789/0 conn(0x7f1f5013f650 legacy=0x7f1f50143a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 >> v1:172.21.15.117:6789/0 conn(0x7f1f500a9d70 legacy=0x7f1f50136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 1 monclient: found mon.c 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f501339d0 con 0x7f1f501457f0 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:14.558 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 client.?.objecter ms_handle_connect 0x7f1f5013f650 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 client.?.objecter ms_handle_connect 0x7f1f500a9d70 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 10 monclient: _renew_subs 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f50144500 con 0x7f1f501457f0 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 10 monclient: _renew_subs 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f501404d0 con 0x7f1f501457f0 2024-03-29T11:25:14.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.573+0000 7f1f57c27640 1 librados: init done 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f57c27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f57c27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f57c27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f50075f10 con 0x7f1f501457f0 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f40001a80 con 0x7f1f501457f0 2024-03-29T11:25:14.561 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f1f400038c0 con 0x7f1f501457f0 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f467fc640 10 monclient: dump: 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:14.562 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.575+0000 7f1f47fff640 4 set_mon_vals no callback set 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.576+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f1f40003ae0 con 0x7f1f501457f0 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f559c3640 10 monclient: get_auth_request con 0x7f1f3406dd70 auth_method 0 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 10 client.4508.objecter ms_handle_connect 0x7f1f3406dd70 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f1f40084a50 con 0x7f1f501457f0 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 10 client.4508.objecter ms_dispatch 0x7f1f50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:14.563 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:14.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.577+0000 7f1f467fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:25:14.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.585+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f40050330 con 0x7f1f501457f0 2024-03-29T11:25:14.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.585+0000 7f1f467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:14.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.585+0000 7f1f467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:14.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.901+0000 7f1f57c27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.902+0000 7f1f57c27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.902+0000 7f1f57c27640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:14.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.902+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f500b3680 con 0x7f1f501457f0 2024-03-29T11:25:14.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.903+0000 7f1f467fc640 1 -- 172.21.15.117:0/1673282869 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f1f40052260 con 0x7f1f501457f0 2024-03-29T11:25:14.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.903+0000 7f1f467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:14.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.903+0000 7f1f467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:14.889 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:14.889 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:14.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.906+0000 7f1f57c27640 10 librados: watch_flush enter 2024-03-29T11:25:14.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.906+0000 7f1f57c27640 10 librados: watch_flush exit 2024-03-29T11:25:14.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.906+0000 7f1f57c27640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:25:14.890 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.906+0000 7f1f57c27640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:25:14.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 >> v1:172.21.15.117:6816/4276038175 conn(0x7f1f3406dd70 legacy=0x7f1f34070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 10 monclient: shutdown 2024-03-29T11:25:14.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:14.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 >> v1:172.21.15.117:6790/0 conn(0x7f1f501457f0 legacy=0x7f1f501ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:14.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 shutdown_connections 2024-03-29T11:25:14.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.907+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 >> 172.21.15.117:0/1673282869 conn(0x7f1f50066a00 msgr2=0x7f1f500b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:14.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.908+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 shutdown_connections 2024-03-29T11:25:14.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.908+0000 7f1f57c27640 1 -- 172.21.15.117:0/1673282869 wait complete. 2024-03-29T11:25:14.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:14.908+0000 7f1f57c27640 1 librados: shutdown 2024-03-29T11:25:14.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:21.716 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.907+0000 7fc2ee8c5640 10 monclient: get_monmap_and_config 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.907+0000 7fc2ee8c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.907+0000 7fc2ee8c5640 10 monclient: monmap: 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.891 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:21.908573+0000 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:21.908573+0000 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.908+0000 7fc2ee8c5640 1 Processor -- start 2024-03-29T11:25:21.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.909+0000 7fc2ee8c5640 1 -- start start 2024-03-29T11:25:21.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.909+0000 7fc2ee8c5640 10 monclient: init 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e00389e0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e0038dc0 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e0030660 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e00c54c0 con 0x7fc2e0038dc0 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e0036430 con 0x7fc2e0030660 2024-03-29T11:25:21.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e0039840 con 0x7fc2e00389e0 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee8c5640 10 monclient(hunting): authenticate will time out at 1190.219368s 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee0c4640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc2e0030660 0x7fc2e00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:51012/0 (socket says 172.21.15.117:51012) 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.910+0000 7fc2ee0c4640 1 -- 172.21.15.117:0/1505635905 learned_addr learned my addr 172.21.15.117:0/1505635905 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783898964 0 0) 0x7fc2e0036430 con 0x7fc2e0030660 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d4003520 con 0x7fc2e0030660 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192770948 0 0) 0x7fc2d4003520 con 0x7fc2e0030660 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 >> v1:172.21.15.117:6790/0 conn(0x7fc2e00389e0 legacy=0x7fc2e0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 >> v1:172.21.15.117:6789/0 conn(0x7fc2e0038dc0 legacy=0x7fc2e00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 monclient: found mon.noname-c 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e0035690 con 0x7fc2e0030660 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc2e804f730 con 0x7fc2e0030660 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ec8c1640 10 monclient: dump: 2024-03-29T11:25:21.899 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ee8c5640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.911+0000 7fc2ee8c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2e80008d0 con 0x7fc2e0030660 2024-03-29T11:25:21.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 1 -- 172.21.15.117:0/1505635905 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc2e804f730 con 0x7fc2e0030660 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ec8c1640 10 monclient: dump: 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:21.901 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ee8c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.912+0000 7fc2ee8c5640 4 set_mon_vals no callback set 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 10 monclient: shutdown 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1505635905 >> v1:172.21.15.173:6789/0 conn(0x7fc2e0030660 legacy=0x7fc2e00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1505635905 shutdown_connections 2024-03-29T11:25:21.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1505635905 >> 172.21.15.117:0/1505635905 conn(0x7fc2e0005c00 msgr2=0x7fc2e0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:21.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1505635905 shutdown_connections 2024-03-29T11:25:21.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.913+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1505635905 wait complete. 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.915+0000 7fc2ee8c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.915+0000 7fc2ee8c5640 10 monclient: monmap: 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:21.909 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.915+0000 7fc2ee8c5640 1 librados: starting msgr at 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.915+0000 7fc2ee8c5640 1 librados: starting objecter 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.916+0000 7fc2ee8c5640 1 Processor -- start 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.916+0000 7fc2ee8c5640 1 -- start start 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.916+0000 7fc2ee8c5640 1 librados: setting wanted keys 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.916+0000 7fc2ee8c5640 1 librados: calling monclient init 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.916+0000 7fc2ee8c5640 10 monclient: init 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e0030660 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e00389e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:21.910 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e00c7c50 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ed0c2640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc2e00389e0 0x7fc2e002f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:51588/0 (socket says 172.21.15.117:51588) 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ed0c2640 1 -- 172.21.15.117:0/1705696654 learned_addr learned my addr 172.21.15.117:0/1705696654 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2de7fc640 10 client.?.objecter ms_handle_connect 0x7fc2e00389e0 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.918+0000 7fc2ee8c5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e002e9e0 con 0x7fc2e00c7c50 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e002f2a0 con 0x7fc2e00389e0 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e002dad0 con 0x7fc2e0030660 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2ee8c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2ee8c5640 10 monclient(hunting): authenticate will time out at 1190.228529s 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 client.?.objecter ms_handle_connect 0x7fc2e0030660 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870063664 0 0) 0x7fc2e002f2a0 con 0x7fc2e00389e0 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.919+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc004eb0 con 0x7fc2e00389e0 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575040573 0 0) 0x7fc2e002dad0 con 0x7fc2e0030660 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e002f2a0 con 0x7fc2e0030660 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 client.?.objecter ms_handle_connect 0x7fc2e00c7c50 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.920+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126326177 0 0) 0x7fc2e002e9e0 con 0x7fc2e00c7c50 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:21.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e002dad0 con 0x7fc2e00c7c50 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171548192 0 0) 0x7fc2cc004eb0 con 0x7fc2e00389e0 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2e002e9e0 con 0x7fc2e00389e0 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc2e4001990 con 0x7fc2e00389e0 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.913 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2303109302 0 0) 0x7fc2e002f2a0 con 0x7fc2e0030660 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2cc004eb0 con 0x7fc2e0030660 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc2d8001820 con 0x7fc2e0030660 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.914 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1293998076 0 0) 0x7fc2e002e9e0 con 0x7fc2e00389e0 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 >> v1:172.21.15.117:6790/0 conn(0x7fc2e0030660 legacy=0x7fc2e002d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 >> v1:172.21.15.117:6789/0 conn(0x7fc2e00c7c50 legacy=0x7fc2e0167220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 monclient: found mon.b 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:21.915 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e002fed0 con 0x7fc2e00389e0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.921+0000 7fc2de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 10 monclient: _renew_subs 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e0168940 con 0x7fc2e00389e0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2e4003610 con 0x7fc2e00389e0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc2e40039e0 con 0x7fc2e00389e0 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:21.916 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2de7fc640 10 monclient: dump: 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 10 monclient: _renew_subs 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:21.917 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e0168cb0 con 0x7fc2e00389e0 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2ee8c5640 1 librados: init done 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.922+0000 7fc2dffff640 4 set_mon_vals no callback set 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.924+0000 7fc2cbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.924+0000 7fc2cbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.924+0000 7fc2cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.924+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e0007b70 con 0x7fc2e00389e0 2024-03-29T11:25:21.918 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.925+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fc2e4003b40 con 0x7fc2e00389e0 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.925+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fc2e4084a90 con 0x7fc2e00389e0 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2de7fc640 10 client.4435.objecter ms_dispatch 0x7fc2e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2de7fc640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2ed8c3640 10 monclient: get_auth_request con 0x7fc2cc06dec0 auth_method 0 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2de7fc640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2de7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T11:25:21.919 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.926+0000 7fc2de7fc640 10 client.4435.objecter ms_handle_connect 0x7fc2cc06dec0 2024-03-29T11:25:21.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.932+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2e4050370 con 0x7fc2e00389e0 2024-03-29T11:25:21.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.932+0000 7fc2de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:21.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:21.932+0000 7fc2de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:22.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.325+0000 7fc2cbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.325+0000 7fc2cbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.309 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.325+0000 7fc2cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:22.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.325+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2e0005120 con 0x7fc2e00389e0 2024-03-29T11:25:22.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.326+0000 7fc2de7fc640 1 -- 172.21.15.117:0/1705696654 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fc2e40522a0 con 0x7fc2e00389e0 2024-03-29T11:25:22.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.326+0000 7fc2de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:22.310 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.326+0000 7fc2de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:22.310 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:22.311 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:22.313 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.329+0000 7fc2cbfff640 10 librados: watch_flush enter 2024-03-29T11:25:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 10 librados: watch_flush exit 2024-03-29T11:25:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T11:25:22.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 10 client.4435.objecter successfully canceled tick 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 >> v1:172.21.15.117:6816/4276038175 conn(0x7fc2cc06dec0 legacy=0x7fc2cc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 10 monclient: shutdown 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.330+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 >> v1:172.21.15.173:6789/0 conn(0x7fc2e00389e0 legacy=0x7fc2e002f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.331+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 shutdown_connections 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.331+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 >> 172.21.15.117:0/1705696654 conn(0x7fc2e0005c00 msgr2=0x7fc2e0005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.331+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 shutdown_connections 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.331+0000 7fc2cbfff640 1 -- 172.21.15.117:0/1705696654 wait complete. 2024-03-29T11:25:22.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:22.331+0000 7fc2cbfff640 1 librados: shutdown 2024-03-29T11:25:22.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:29.138 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.324+0000 7f673657b640 10 monclient: get_monmap_and_config 2024-03-29T11:25:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.324+0000 7f673657b640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.308 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.324+0000 7f673657b640 10 monclient: monmap: 2024-03-29T11:25:29.308 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:29.325137+0000 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:29.325137+0000 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:29.309 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.325+0000 7f673657b640 1 Processor -- start 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.325+0000 7f673657b640 1 -- start start 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.325+0000 7f673657b640 10 monclient: init 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient(hunting): picked mon.noname-b con 0x7f67300b2790 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:29.311 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient(hunting): picked mon.noname-a con 0x7f67300b37d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient(hunting): picked mon.noname-c con 0x7f6730146940 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300b2b70 con 0x7f67300b37d0 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673006eec0 con 0x7f6730146940 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673006f1a0 con 0x7f67300b2790 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.326+0000 7f673657b640 10 monclient(hunting): authenticate will time out at 1197.635154s 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.328+0000 7f672f7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f67300b37d0 0x7f6730146230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:40184/0 (socket says 172.21.15.117:40184) 2024-03-29T11:25:29.312 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.328+0000 7f672f7fe640 1 -- 172.21.15.117:0/3624769015 learned_addr learned my addr 172.21.15.117:0/3624769015 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:29.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.330+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7585889 0 0) 0x7f673006f1a0 con 0x7f67300b2790 2024-03-29T11:25:29.314 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.330+0000 7f672effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.330+0000 7f672effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.330+0000 7f672effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.330+0000 7f672effd640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:25:29.315 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720003520 con 0x7f67300b2790 2024-03-29T11:25:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459037300 0 0) 0x7f6720003520 con 0x7f67300b2790 2024-03-29T11:25:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 >> v1:172.21.15.173:6789/0 conn(0x7f6730146940 legacy=0x7f6730151b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:25:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 >> v1:172.21.15.117:6789/0 conn(0x7f67300b37d0 legacy=0x7f6730146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.316 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 monclient: found mon.noname-b 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67300b2e20 con 0x7f67300b2790 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f673657b640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f673657b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6724001690 con 0x7f67300b2790 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f672effd640 10 monclient: dump: 2024-03-29T11:25:29.317 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.331+0000 7f673657b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6724001920 con 0x7f67300b2790 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:29.318 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 4 set_mon_vals no callback set 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 1 -- 172.21.15.117:0/3624769015 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6724001cb0 con 0x7f67300b2790 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f672effd640 10 monclient: dump: 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:29.319 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 10 monclient: shutdown 2024-03-29T11:25:29.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 1 -- 172.21.15.117:0/3624769015 >> v1:172.21.15.117:6790/0 conn(0x7f67300b2790 legacy=0x7f6730143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 1 -- 172.21.15.117:0/3624769015 shutdown_connections 2024-03-29T11:25:29.320 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 1 -- 172.21.15.117:0/3624769015 >> 172.21.15.117:0/3624769015 conn(0x7f673006d180 msgr2=0x7f6730066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.332+0000 7f673657b640 1 -- 172.21.15.117:0/3624769015 shutdown_connections 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.333+0000 7f673657b640 1 -- 172.21.15.117:0/3624769015 wait complete. 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 10 monclient: build_initial_monmap 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 10 monclient: monmap: 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:29.321 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 1 librados: starting msgr at 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 1 librados: starting objecter 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 1 Processor -- start 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.334+0000 7f673657b640 1 -- start start 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 1 librados: setting wanted keys 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 1 librados: calling monclient init 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient: init 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:29.322 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient(hunting): picked mon.noname-a con 0x7f67300b2790 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient(hunting): picked mon.noname-b con 0x7f67300b37d0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient(hunting): picked mon.noname-c con 0x7f6730146940 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673014a480 con 0x7f67300b2790 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6730138cb0 con 0x7f67300b37d0 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6730149810 con 0x7f6730146940 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:29.323 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.335+0000 7f673657b640 10 monclient(hunting): authenticate will time out at 1197.644850s 2024-03-29T11:25:29.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.338+0000 7f672ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f67300b2790 0x7f6730148fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42774/0 (socket says 172.21.15.117:42774) 2024-03-29T11:25:29.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.338+0000 7f672ffff640 1 -- 172.21.15.117:0/2397970174 learned_addr learned my addr 172.21.15.117:0/2397970174 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:29.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 client.?.objecter ms_handle_connect 0x7f67300b2790 2024-03-29T11:25:29.325 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299989674 0 0) 0x7f673014a480 con 0x7f67300b2790 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.339+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714004eb0 con 0x7f67300b2790 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430952374 0 0) 0x7f6714004eb0 con 0x7f67300b2790 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f673014a480 con 0x7f67300b2790 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6724001690 con 0x7f67300b2790 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 10 monclient(hunting): dump: 2024-03-29T11:25:29.326 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463540807 0 0) 0x7f673014a480 con 0x7f67300b2790 2024-03-29T11:25:29.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.340+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 >> v1:172.21.15.117:6790/0 conn(0x7f6730146940 legacy=0x7f6730137f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:29.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 >> v1:172.21.15.173:6789/0 conn(0x7f67300b37d0 legacy=0x7f6730139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:29.328 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 1 monclient: found mon.a 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730139850 con 0x7f67300b2790 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 client.?.objecter ms_handle_connect 0x7f67300b37d0 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f672cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 10 monclient: _renew_subs 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6730138680 con 0x7f67300b2790 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:29.329 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 10 monclient: _renew_subs 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6730148530 con 0x7f67300b2790 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.343+0000 7f673657b640 1 librados: init done 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.344+0000 7f673657b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.344+0000 7f673657b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.344+0000 7f673657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:25:29.330 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.344+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67301426c0 con 0x7f67300b2790 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67240038a0 con 0x7f67300b2790 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6724003cb0 con 0x7f67300b2790 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.345+0000 7f672cff9640 10 monclient: dump: 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:29.332 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.346+0000 7f672dffb640 4 set_mon_vals no callback set 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.346+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f6724003ed0 con 0x7f67300b2790 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f6724084b70 con 0x7f67300b2790 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 10 client.4434.objecter ms_dispatch 0x7f6730000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672f7fe640 10 monclient: get_auth_request con 0x7f671406dc00 auth_method 0 2024-03-29T11:25:29.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.347+0000 7f672cff9640 10 client.4434.objecter ms_handle_connect 0x7f671406dc00 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.357+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6724050450 con 0x7f67300b2790 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.357+0000 7f672cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:29.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.357+0000 7f672cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:29.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.702+0000 7f673657b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.702+0000 7f673657b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.702+0000 7f673657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:25:29.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.702+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67300008d0 con 0x7f67300b2790 2024-03-29T11:25:29.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.704+0000 7f672cff9640 1 -- 172.21.15.117:0/2397970174 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f6724052380 con 0x7f67300b2790 2024-03-29T11:25:29.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.704+0000 7f672cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:29.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.704+0000 7f672cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:29.688 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:29.689 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:29.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 10 librados: watch_flush enter 2024-03-29T11:25:29.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 10 librados: watch_flush exit 2024-03-29T11:25:29.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:25:29.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:25:29.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 >> v1:172.21.15.117:6816/4276038175 conn(0x7f671406dc00 legacy=0x7f67140700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 10 monclient: shutdown 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 >> v1:172.21.15.117:6789/0 conn(0x7f67300b2790 legacy=0x7f6730148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 shutdown_connections 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.709+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 >> 172.21.15.117:0/2397970174 conn(0x7f673006d180 msgr2=0x7f67300b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.710+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 shutdown_connections 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.710+0000 7f673657b640 1 -- 172.21.15.117:0/2397970174 wait complete. 2024-03-29T11:25:29.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:29.710+0000 7f673657b640 1 librados: shutdown 2024-03-29T11:25:29.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:36.518 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.678+0000 7fccbc2ba640 10 monclient: get_monmap_and_config 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.678+0000 7fccbc2ba640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.678+0000 7fccbc2ba640 10 monclient: monmap: 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:36.679905+0000 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:36.679905+0000 2024-03-29T11:25:36.663 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.679+0000 7fccbc2ba640 1 Processor -- start 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.679+0000 7fccbc2ba640 1 -- start start 2024-03-29T11:25:36.664 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.679+0000 7fccbc2ba640 10 monclient: init 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.680+0000 7fccbc2ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.680+0000 7fccbc2ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.680+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-b con 0x7fccb41422b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-a con 0x7fccb4142690 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-c con 0x7fccb40b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb4142c80 con 0x7fccb4142690 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40aae40 con 0x7fccb40b17d0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb413fbd0 con 0x7fccb41422b0 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.666 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccbc2ba640 10 monclient(hunting): authenticate will time out at 1204.990061s 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccba056640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fccb41422b0 0x7fccb413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:60434/0 (socket says 172.21.15.117:60434) 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.681+0000 7fccba056640 1 -- 172.21.15.117:0/3909687973 learned_addr learned my addr 172.21.15.117:0/3909687973 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.682+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090483315 0 0) 0x7fccb413fbd0 con 0x7fccb41422b0 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca0003520 con 0x7fccb41422b0 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572614707 0 0) 0x7fcca0003520 con 0x7fccb41422b0 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.683+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 >> v1:172.21.15.173:6789/0 conn(0x7fccb40b17d0 legacy=0x7fccb40b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:36.668 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.684+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 >> v1:172.21.15.117:6789/0 conn(0x7fccb4142690 legacy=0x7fccb40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 1 monclient: found mon.noname-b 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4151940 con 0x7fccb41422b0 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.673 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fccac001a10 con 0x7fccb41422b0 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 10 monclient: dump: 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:36.674 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.687+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccac001c30 con 0x7fccb41422b0 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccb9054640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccb9054640 1 -- 172.21.15.117:0/3909687973 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fccac001fc0 con 0x7fccb41422b0 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccbc2ba640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccbc2ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccbc2ba640 4 set_mon_vals no callback set 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccb9054640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccb9054640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.688+0000 7fccb9054640 10 monclient: dump: 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.675 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 10 monclient: shutdown 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 1 -- 172.21.15.117:0/3909687973 >> v1:172.21.15.117:6790/0 conn(0x7fccb41422b0 legacy=0x7fccb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 1 -- 172.21.15.117:0/3909687973 shutdown_connections 2024-03-29T11:25:36.676 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 1 -- 172.21.15.117:0/3909687973 >> 172.21.15.117:0/3909687973 conn(0x7fccb4066a00 msgr2=0x7fccb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:36.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 1 -- 172.21.15.117:0/3909687973 shutdown_connections 2024-03-29T11:25:36.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.689+0000 7fccbc2ba640 1 -- 172.21.15.117:0/3909687973 wait complete. 2024-03-29T11:25:36.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.690+0000 7fccbc2ba640 10 monclient: build_initial_monmap 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.690+0000 7fccbc2ba640 10 monclient: monmap: 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 librados: starting msgr at 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 librados: starting objecter 2024-03-29T11:25:36.678 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 Processor -- start 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 -- start start 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 librados: setting wanted keys 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 1 librados: calling monclient init 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.691+0000 7fccbc2ba640 10 monclient: init 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-c con 0x7fccb40b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-b con 0x7fccb41422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient(hunting): picked mon.noname-a con 0x7fccb4142690 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb412fba0 con 0x7fccb4142690 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb406f7d0 con 0x7fccb41422b0 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb4131ec0 con 0x7fccb40b17d0 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:36.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.693+0000 7fccbc2ba640 10 monclient(hunting): authenticate will time out at 1205.002380s 2024-03-29T11:25:36.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.695+0000 7fccba056640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fccb40b17d0 0x7fccb406e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37370/0 (socket says 172.21.15.117:37370) 2024-03-29T11:25:36.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.695+0000 7fccba056640 1 -- 172.21.15.117:0/973771099 learned_addr learned my addr 172.21.15.117:0/973771099 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 client.?.objecter ms_handle_connect 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189503867 0 0) 0x7fccb4131ec0 con 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc98004eb0 con 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274865411 0 0) 0x7fcc98004eb0 con 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.696+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccb4131ec0 con 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fccac0018d0 con 0x7fccb40b17d0 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:36.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 client.?.objecter ms_handle_connect 0x7fccb41422b0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172133647 0 0) 0x7fccb406f7d0 con 0x7fccb41422b0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc98004eb0 con 0x7fccb41422b0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2162862227 0 0) 0x7fccb4131ec0 con 0x7fccb40b17d0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 >> v1:172.21.15.173:6789/0 conn(0x7fccb41422b0 legacy=0x7fccb4130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 >> v1:172.21.15.117:6789/0 conn(0x7fccb4142690 legacy=0x7fccb4130740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.697+0000 7fcca6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 1 monclient: found mon.c 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:36.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4132680 con 0x7fccb40b17d0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fcca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 10 monclient: _renew_subs 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb41328d0 con 0x7fccb40b17d0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 10 monclient: _renew_subs 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb41efc30 con 0x7fccb40b17d0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.698+0000 7fccbc2ba640 1 librados: init done 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccac001a30 con 0x7fccb40b17d0 2024-03-29T11:25:36.688 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fccac005110 con 0x7fccb40b17d0 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.699+0000 7fcca6ffd640 10 monclient: dump: 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:36.689 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.700+0000 7fccb8853640 4 set_mon_vals no callback set 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.701+0000 7fccbc2ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.701+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fccac01b0e0 con 0x7fccb40b17d0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.701+0000 7fccbc2ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.701+0000 7fccbc2ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.701+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c005080 con 0x7fccb40b17d0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fccac085080 con 0x7fccb40b17d0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 10 client.4523.objecter ms_dispatch 0x7fccb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:25:36.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fccb9855640 10 monclient: get_auth_request con 0x7fcc9806ddb0 auth_method 0 2024-03-29T11:25:36.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.702+0000 7fcca6ffd640 10 client.4523.objecter ms_handle_connect 0x7fcc9806ddb0 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.711+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccac050960 con 0x7fccb40b17d0 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.711+0000 7fcca6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:36.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:36.711+0000 7fcca6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.038+0000 7fccbc2ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.038+0000 7fccbc2ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.038+0000 7fccbc2ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:37.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.038+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc9c005330 con 0x7fccb40b17d0 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.040+0000 7fcca6ffd640 1 -- 172.21.15.117:0/973771099 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fccac052890 con 0x7fccb40b17d0 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.040+0000 7fcca6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:37.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.040+0000 7fcca6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:37.026 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:37.026 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:37.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.044+0000 7fccbc2ba640 10 librados: watch_flush enter 2024-03-29T11:25:37.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.044+0000 7fccbc2ba640 10 librados: watch_flush exit 2024-03-29T11:25:37.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.044+0000 7fccbc2ba640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.044+0000 7fccbc2ba640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 >> v1:172.21.15.117:6816/4276038175 conn(0x7fcc9806ddb0 legacy=0x7fcc98070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 10 monclient: shutdown 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 >> v1:172.21.15.117:6790/0 conn(0x7fccb40b17d0 legacy=0x7fccb406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 shutdown_connections 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 >> 172.21.15.117:0/973771099 conn(0x7fccb4066a00 msgr2=0x7fccb40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 shutdown_connections 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 -- 172.21.15.117:0/973771099 wait complete. 2024-03-29T11:25:37.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:37.045+0000 7fccbc2ba640 1 librados: shutdown 2024-03-29T11:25:37.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:43.853 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.013+0000 7fbae1982640 10 monclient: get_monmap_and_config 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.013+0000 7fbae1982640 10 monclient: build_initial_monmap 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.013+0000 7fbae1982640 10 monclient: monmap: 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:44.014479+0000 2024-03-29T11:25:43.998 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:44.014479+0000 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:43.999 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.014+0000 7fbae1982640 1 Processor -- start 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.014+0000 7fbae1982640 1 -- start start 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.014+0000 7fbae1982640 10 monclient: init 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc13f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbada7fc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fbadc13f650 0x7fbadc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:39364/0 (socket says 172.21.15.117:39364) 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbada7fc640 1 -- 172.21.15.117:0/3477097094 learned_addr learned my addr 172.21.15.117:0/3477097094 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:44.005 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.015+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc0aaef0 con 0x7fbadc0a9d70 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc145bd0 con 0x7fbadc13f650 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc145eb0 con 0x7fbadc1457f0 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbae1982640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbae1982640 10 monclient(hunting): authenticate will time out at 1212.325171s 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629030285 0 0) 0x7fbadc0aaef0 con 0x7fbadc0a9d70 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac4003520 con 0x7fbadc0a9d70 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258697843 0 0) 0x7fbadc145bd0 con 0x7fbadc13f650 2024-03-29T11:25:44.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.016+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc0aaef0 con 0x7fbadc13f650 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581301825 0 0) 0x7fbac4003520 con 0x7fbadc0a9d70 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 >> v1:172.21.15.117:6790/0 conn(0x7fbadc1457f0 legacy=0x7fbadc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 >> v1:172.21.15.173:6789/0 conn(0x7fbadc13f650 legacy=0x7fbadc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc146340 con 0x7fbadc0a9d70 2024-03-29T11:25:44.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbad4001a10 con 0x7fbadc0a9d70 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: dump: 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:44.008 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad4001b70 con 0x7fbadc0a9d70 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 1 -- 172.21.15.117:0/3477097094 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbad4001f80 con 0x7fbadc0a9d70 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.017+0000 7fbad9ffb640 10 monclient: dump: 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:44.009 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.018+0000 7fbae1982640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.018+0000 7fbae1982640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.018+0000 7fbae1982640 4 set_mon_vals no callback set 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.018+0000 7fbae1982640 10 monclient: shutdown 2024-03-29T11:25:44.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 >> v1:172.21.15.117:6789/0 conn(0x7fbadc0a9d70 legacy=0x7fbadc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 shutdown_connections 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 >> 172.21.15.117:0/3477097094 conn(0x7fbadc066a00 msgr2=0x7fbadc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 shutdown_connections 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.019+0000 7fbae1982640 1 -- 172.21.15.117:0/3477097094 wait complete. 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.020+0000 7fbae1982640 10 monclient: build_initial_monmap 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.020+0000 7fbae1982640 10 monclient: monmap: 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:44.011 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:44.012 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 librados: starting msgr at 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 librados: starting objecter 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 Processor -- start 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 -- start start 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 librados: setting wanted keys 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 1 librados: calling monclient init 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.021+0000 7fbae1982640 10 monclient: init 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:44.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc13f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc13b210 con 0x7fbadc0a9d70 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc1cb340 con 0x7fbadc1457f0 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbadc13b460 con 0x7fbadc13f650 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbae1982640 10 monclient(hunting): authenticate will time out at 1212.331650s 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbadb7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fbadc1457f0 0x7fbadc1e6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:39772/0 (socket says 172.21.15.117:39772) 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.022+0000 7fbadb7fe640 1 -- 172.21.15.117:0/1462874244 learned_addr learned my addr 172.21.15.117:0/1462874244 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 client.?.objecter ms_handle_connect 0x7fbadc1457f0 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164741543 0 0) 0x7fbadc1cb340 con 0x7fbadc1457f0 2024-03-29T11:25:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:25:44.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac8004eb0 con 0x7fbadc1457f0 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 client.?.objecter ms_handle_connect 0x7fbadc13f650 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1829490227 0 0) 0x7fbac8004eb0 con 0x7fbadc1457f0 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbadc1cb340 con 0x7fbadc1457f0 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbad4001ec0 con 0x7fbadc1457f0 2024-03-29T11:25:44.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): dump: 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:44.022 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.023+0000 7fbac3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 client.?.objecter ms_handle_connect 0x7fbadc0a9d70 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2708603516 0 0) 0x7fbadc1cb340 con 0x7fbadc1457f0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 >> v1:172.21.15.117:6790/0 conn(0x7fbadc13f650 legacy=0x7fbadc1e36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 >> v1:172.21.15.117:6789/0 conn(0x7fbadc0a9d70 legacy=0x7fbadc140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 1 monclient: found mon.b 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:44.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc135ad0 con 0x7fbadc1457f0 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 10 monclient: _renew_subs 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc138e00 con 0x7fbadc1457f0 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 10 monclient: _renew_subs 2024-03-29T11:25:44.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc13bb30 con 0x7fbadc1457f0 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.024+0000 7fbae1982640 1 librados: init done 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad4003c20 con 0x7fbadc1457f0 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbad97fa640 4 set_mon_vals no callback set 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbad4004030 con 0x7fbadc1457f0 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.025+0000 7fbac3fff640 10 monclient: dump: 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:44.025 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.026+0000 7fbae1982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.026+0000 7fbae1982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.026+0000 7fbae1982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:44.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.026+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbabc005080 con 0x7fbadc1457f0 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.027+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fbad4004210 con 0x7fbadc1457f0 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fbad4084fb0 con 0x7fbadc1457f0 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 10 client.4447.objecter ms_dispatch 0x7fbadc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbadaffd640 10 monclient: get_auth_request con 0x7fbac806dc70 auth_method 0 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.028+0000 7fbac3fff640 10 client.4447.objecter ms_handle_connect 0x7fbac806dc70 2024-03-29T11:25:44.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.034+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbad4050890 con 0x7fbadc1457f0 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.034+0000 7fbac3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:44.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.034+0000 7fbac3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:44.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.409+0000 7fbae1982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.409+0000 7fbae1982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.409+0000 7fbae1982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:44.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.409+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbabc005330 con 0x7fbadc1457f0 2024-03-29T11:25:44.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.411+0000 7fbac3fff640 1 -- 172.21.15.117:0/1462874244 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fbad40527c0 con 0x7fbadc1457f0 2024-03-29T11:25:44.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.411+0000 7fbac3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:44.396 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.411+0000 7fbac3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:44.396 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:44.396 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:44.398 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 10 librados: watch_flush enter 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 10 librados: watch_flush exit 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 >> v1:172.21.15.117:6816/4276038175 conn(0x7fbac806dc70 legacy=0x7fbac8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 10 monclient: shutdown 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:44.399 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 >> v1:172.21.15.173:6789/0 conn(0x7fbadc1457f0 legacy=0x7fbadc1e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:44.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 shutdown_connections 2024-03-29T11:25:44.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.414+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 >> 172.21.15.117:0/1462874244 conn(0x7fbadc066a00 msgr2=0x7fbadc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:44.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.415+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 shutdown_connections 2024-03-29T11:25:44.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.415+0000 7fbae1982640 1 -- 172.21.15.117:0/1462874244 wait complete. 2024-03-29T11:25:44.400 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:44.415+0000 7fbae1982640 1 librados: shutdown 2024-03-29T11:25:44.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:51.222 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.379+0000 7f6c89a82640 10 monclient: get_monmap_and_config 2024-03-29T11:25:51.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.379+0000 7f6c89a82640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.379+0000 7f6c89a82640 10 monclient: monmap: 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:51.380953+0000 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:51.380953+0000 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:51.365 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.384+0000 7f6c89a82640 1 Processor -- start 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.384+0000 7f6c89a82640 1 -- start start 2024-03-29T11:25:51.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.384+0000 7f6c89a82640 10 monclient: init 2024-03-29T11:25:51.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-b con 0x7f6c841457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8413f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-c con 0x7f6c840a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840aaef0 con 0x7f6c8413f650 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c84145bd0 con 0x7f6c840a9d70 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c84145eb0 con 0x7f6c841457f0 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.385+0000 7f6c89a82640 10 monclient(hunting): authenticate will time out at 1219.694749s 2024-03-29T11:25:51.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.387+0000 7f6c83fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6c840a9d70 0x7f6c8414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53106/0 (socket says 172.21.15.117:53106) 2024-03-29T11:25:51.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.387+0000 7f6c83fff640 1 -- 172.21.15.117:0/4196591622 learned_addr learned my addr 172.21.15.117:0/4196591622 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567770577 0 0) 0x7f6c84145bd0 con 0x7f6c840a9d70 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.376 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c003520 con 0x7f6c840a9d70 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267424860 0 0) 0x7f6c84145eb0 con 0x7f6c841457f0 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c003840 con 0x7f6c841457f0 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.389+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746879362 0 0) 0x7f6c840aaef0 con 0x7f6c8413f650 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.377 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c84145eb0 con 0x7f6c8413f650 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280562338 0 0) 0x7f6c6c003840 con 0x7f6c841457f0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 >> v1:172.21.15.173:6789/0 conn(0x7f6c840a9d70 legacy=0x7f6c8414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 >> v1:172.21.15.117:6789/0 conn(0x7f6c8413f650 legacy=0x7f6c840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 monclient: found mon.noname-b 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c84146340 con 0x7f6c841457f0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c780015a0 con 0x7f6c841457f0 2024-03-29T11:25:51.378 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.390+0000 7f6c827fc640 10 monclient: dump: 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c78001700 con 0x7f6c841457f0 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.379 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 1 -- 172.21.15.117:0/4196591622 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c78001b10 con 0x7f6c841457f0 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c827fc640 10 monclient: dump: 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:51.380 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c89a82640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c89a82640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:51.382 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.391+0000 7f6c89a82640 4 set_mon_vals no callback set 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 10 monclient: shutdown 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 1 -- 172.21.15.117:0/4196591622 >> v1:172.21.15.117:6790/0 conn(0x7f6c841457f0 legacy=0x7f6c8413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 1 -- 172.21.15.117:0/4196591622 shutdown_connections 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 1 -- 172.21.15.117:0/4196591622 >> 172.21.15.117:0/4196591622 conn(0x7f6c84066a00 msgr2=0x7f6c84066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 1 -- 172.21.15.117:0/4196591622 shutdown_connections 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.393+0000 7f6c89a82640 1 -- 172.21.15.117:0/4196591622 wait complete. 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.394+0000 7f6c89a82640 10 monclient: build_initial_monmap 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.394+0000 7f6c89a82640 10 monclient: monmap: 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.383 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 librados: starting msgr at 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 librados: starting objecter 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 Processor -- start 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 -- start start 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 librados: setting wanted keys 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 1 librados: calling monclient init 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.395+0000 7f6c89a82640 10 monclient: init 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-a con 0x7f6c840a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:51.384 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-c con 0x7f6c8413f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient(hunting): picked mon.noname-b con 0x7f6c841457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8413fe30 con 0x7f6c840a9d70 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c841cc9a0 con 0x7f6c841457f0 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c84138780 con 0x7f6c8413f650 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c89a82640 10 monclient(hunting): authenticate will time out at 1219.705572s 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c83fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6c841457f0 0x7f6c841e6270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53840/0 (socket says 172.21.15.117:53840) 2024-03-29T11:25:51.385 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.396+0000 7f6c83fff640 1 -- 172.21.15.117:0/299691447 learned_addr learned my addr 172.21.15.117:0/299691447 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 client.?.objecter ms_handle_connect 0x7f6c841457f0 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859354757 0 0) 0x7f6c841cc9a0 con 0x7f6c841457f0 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.387 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c70004eb0 con 0x7f6c841457f0 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.399+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952461921 0 0) 0x7f6c70004eb0 con 0x7f6c841457f0 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c841cc9a0 con 0x7f6c841457f0 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c7c001cb0 con 0x7f6c841457f0 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.388 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 client.?.objecter ms_handle_connect 0x7f6c840a9d70 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216968531 0 0) 0x7f6c8413fe30 con 0x7f6c840a9d70 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c70004eb0 con 0x7f6c840a9d70 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 144214711 0 0) 0x7f6c841cc9a0 con 0x7f6c841457f0 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 >> v1:172.21.15.117:6790/0 conn(0x7f6c8413f650 legacy=0x7f6c841409d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:51.389 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 >> v1:172.21.15.117:6789/0 conn(0x7f6c840a9d70 legacy=0x7f6c841402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 monclient: found mon.b 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c841422b0 con 0x7f6c841457f0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 10 monclient: _renew_subs 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c841425d0 con 0x7f6c841457f0 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 10 monclient: _renew_subs 2024-03-29T11:25:51.390 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c841cb790 con 0x7f6c841457f0 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.400+0000 7f6c89a82640 1 librados: init done 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.402+0000 7f6c89a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.402+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c7c004b60 con 0x7f6c841457f0 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.402+0000 7f6c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.402+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f6c7c004e70 con 0x7f6c841457f0 2024-03-29T11:25:51.391 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.402+0000 7f6c81ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.403+0000 7f6c89a82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.403+0000 7f6c89a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.403+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c84142920 con 0x7f6c841457f0 2024-03-29T11:25:51.392 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.404+0000 7f6c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.404+0000 7f6c6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.404+0000 7f6c6bfff640 10 monclient: dump: 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.406+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f6c7c01ae40 con 0x7f6c841457f0 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f6c7c084d20 con 0x7f6c841457f0 2024-03-29T11:25:51.393 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c6bfff640 10 client.4453.objecter ms_dispatch 0x7f6c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:51.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c6bfff640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:51.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c6bfff640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:51.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c6bfff640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:25:51.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.407+0000 7f6c837fe640 10 monclient: get_auth_request con 0x7f6c7006dc60 auth_method 0 2024-03-29T11:25:51.394 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.409+0000 7f6c6bfff640 10 client.4453.objecter ms_handle_connect 0x7f6c7006dc60 2024-03-29T11:25:51.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.415+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c7c050430 con 0x7f6c841457f0 2024-03-29T11:25:51.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.415+0000 7f6c6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:51.402 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.415+0000 7f6c6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:51.719 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.734+0000 7f6c89a82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.734+0000 7f6c89a82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.734+0000 7f6c89a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:25:51.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.734+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c84075f10 con 0x7f6c841457f0 2024-03-29T11:25:51.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.736+0000 7f6c6bfff640 1 -- 172.21.15.117:0/299691447 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f6c7c052530 con 0x7f6c841457f0 2024-03-29T11:25:51.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.736+0000 7f6c6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:51.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.736+0000 7f6c6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:51.721 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:51.721 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:51.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.740+0000 7f6c89a82640 10 librados: watch_flush enter 2024-03-29T11:25:51.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.740+0000 7f6c89a82640 10 librados: watch_flush exit 2024-03-29T11:25:51.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.740+0000 7f6c89a82640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:25:51.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.740+0000 7f6c89a82640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:25:51.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.740+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 >> v1:172.21.15.117:6816/4276038175 conn(0x7f6c7006dc60 legacy=0x7f6c70070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.741+0000 7f6c89a82640 10 monclient: shutdown 2024-03-29T11:25:51.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.741+0000 7f6c89a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:51.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.741+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 >> v1:172.21.15.173:6789/0 conn(0x7f6c841457f0 legacy=0x7f6c841e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:51.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.741+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 shutdown_connections 2024-03-29T11:25:51.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.741+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 >> 172.21.15.117:0/299691447 conn(0x7f6c84066a00 msgr2=0x7f6c8413b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:51.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.744+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 shutdown_connections 2024-03-29T11:25:51.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.744+0000 7f6c89a82640 1 -- 172.21.15.117:0/299691447 wait complete. 2024-03-29T11:25:51.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:51.744+0000 7f6c89a82640 1 librados: shutdown 2024-03-29T11:25:51.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:58.550 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.691+0000 7f563114d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.691+0000 7f563114d640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.691+0000 7f563114d640 10 monclient: monmap: 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:25:58.692871+0000 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:25:58.692871+0000 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.677 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.678 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:58.678 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:25:58.678 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:25:58.678 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.692+0000 7f563114d640 1 Processor -- start 2024-03-29T11:25:58.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.692+0000 7f563114d640 1 -- start start 2024-03-29T11:25:58.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.692+0000 7f563114d640 10 monclient: init 2024-03-29T11:25:58.679 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient(hunting): picked mon.noname-a con 0x7f562c071de0 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient(hunting): picked mon.noname-c con 0x7f562c0721c0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient(hunting): picked mon.noname-b con 0x7f562c145ae0 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c145650 con 0x7f562c071de0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c0ae260 con 0x7f562c0721c0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c1420d0 con 0x7f562c145ae0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f563114d640 10 monclient(hunting): authenticate will time out at 1227.002835s 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f562b59e640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f562c145ae0 0x7f562c145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40402/0 (socket says 172.21.15.117:40402) 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.693+0000 7f562b59e640 1 -- 172.21.15.117:0/3657701838 learned_addr learned my addr 172.21.15.117:0/3657701838 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136678963 0 0) 0x7f562c1420d0 con 0x7f562c145ae0 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5618003520 con 0x7f562c145ae0 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537198275 0 0) 0x7f5618003520 con 0x7f562c145ae0 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 >> v1:172.21.15.173:6789/0 conn(0x7f562c0721c0 legacy=0x7f562c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.694+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 >> v1:172.21.15.117:6789/0 conn(0x7f562c071de0 legacy=0x7f562c0aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 1 monclient: found mon.noname-b 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0b0590 con 0x7f562c145ae0 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.684 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.698+0000 7f562a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5624001a10 con 0x7f562c145ae0 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: dump: 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:58.685 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5624001c30 con 0x7f562c145ae0 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 1 -- 172.21.15.117:0/3657701838 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5624001fc0 con 0x7f562c145ae0 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f562a59c640 10 monclient: dump: 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.686 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f563114d640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f563114d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:58.687 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.699+0000 7f563114d640 4 set_mon_vals no callback set 2024-03-29T11:25:58.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 10 monclient: shutdown 2024-03-29T11:25:58.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:58.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 1 -- 172.21.15.117:0/3657701838 >> v1:172.21.15.117:6790/0 conn(0x7f562c145ae0 legacy=0x7f562c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 1 -- 172.21.15.117:0/3657701838 shutdown_connections 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 1 -- 172.21.15.117:0/3657701838 >> 172.21.15.117:0/3657701838 conn(0x7f562c06b660 msgr2=0x7f562c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 1 -- 172.21.15.117:0/3657701838 shutdown_connections 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.700+0000 7f563114d640 1 -- 172.21.15.117:0/3657701838 wait complete. 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.701+0000 7f563114d640 10 monclient: build_initial_monmap 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.701+0000 7f563114d640 10 monclient: monmap: 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.695 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.701+0000 7f563114d640 1 librados: starting msgr at 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.701+0000 7f563114d640 1 librados: starting objecter 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.701+0000 7f563114d640 1 Processor -- start 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.702+0000 7f563114d640 1 -- start start 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.702+0000 7f563114d640 1 librados: setting wanted keys 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.702+0000 7f563114d640 1 librados: calling monclient init 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.702+0000 7f563114d640 10 monclient: init 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 10 monclient(hunting): picked mon.noname-a con 0x7f562c071de0 addr v1:172.21.15.117:6789/0 2024-03-29T11:25:58.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 10 monclient(hunting): picked mon.noname-b con 0x7f562c145ae0 addr v1:172.21.15.173:6789/0 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f5623fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f562c145ae0 0x7f562c137740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:58312/0 (socket says 172.21.15.117:58312) 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f5623fff640 1 -- 172.21.15.117:0/1086880080 learned_addr learned my addr 172.21.15.117:0/1086880080 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 10 monclient(hunting): picked mon.noname-c con 0x7f562c137e50 addr v1:172.21.15.117:6790/0 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c071ac0 con 0x7f562c071de0 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f56237fe640 10 client.?.objecter ms_handle_connect 0x7f562c137e50 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f56237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.703+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c06f5d0 con 0x7f562c145ae0 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c070940 con 0x7f562c137e50 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f563114d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f563114d640 10 monclient(hunting): authenticate will time out at 1227.013214s 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227544122 0 0) 0x7f562c070940 con 0x7f562c137e50 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:58.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c004eb0 con 0x7f562c137e50 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580290038 0 0) 0x7f560c004eb0 con 0x7f562c137e50 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f560c005300 con 0x7f562c137e50 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5624001730 con 0x7f562c137e50 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): dump: 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.698 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2495814023 0 0) 0x7f560c005300 con 0x7f562c137e50 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 >> v1:172.21.15.173:6789/0 conn(0x7f562c145ae0 legacy=0x7f562c137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.704+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 >> v1:172.21.15.117:6789/0 conn(0x7f562c071de0 legacy=0x7f562c0713b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 1 monclient: found mon.c 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c070c20 con 0x7f562c137e50 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:58.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 10 monclient: _renew_subs 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1360b0 con 0x7f562c137e50 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 10 monclient: _renew_subs 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1e7640 con 0x7f562c137e50 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5624004f70 con 0x7f562c137e50 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f563114d640 1 librados: init done 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f5624001ad0 con 0x7f562c137e50 2024-03-29T11:25:58.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f56237fe640 10 monclient: dump: 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.705+0000 7f5629d9b640 4 set_mon_vals no callback set 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.707+0000 7f563114d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.707+0000 7f563114d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.707+0000 7f563114d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.707+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c075f10 con 0x7f562c137e50 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.708+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f562401b0f0 con 0x7f562c137e50 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f5624084f10 con 0x7f562c137e50 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f56237fe640 10 client.4538.objecter ms_dispatch 0x7f562c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f56237fe640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f56237fe640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f56237fe640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:25:58.702 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.709+0000 7f562ad9d640 10 monclient: get_auth_request con 0x7f560c06e0e0 auth_method 0 2024-03-29T11:25:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.710+0000 7f56237fe640 10 client.4538.objecter ms_handle_connect 0x7f560c06e0e0 2024-03-29T11:25:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.715+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56240507f0 con 0x7f562c137e50 2024-03-29T11:25:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.715+0000 7f56237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:58.703 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:58.715+0000 7f56237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:59.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.033+0000 7f563114d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.033+0000 7f563114d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.033+0000 7f563114d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:25:59.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.033+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f562c06f7d0 con 0x7f562c137e50 2024-03-29T11:25:59.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.034+0000 7f56237fe640 1 -- 172.21.15.117:0/1086880080 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f56240507f0 con 0x7f562c137e50 2024-03-29T11:25:59.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.034+0000 7f56237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:59.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.034+0000 7f56237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:59.020 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:25:59.020 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 10 librados: watch_flush enter 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 10 librados: watch_flush exit 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 >> v1:172.21.15.117:6816/4276038175 conn(0x7f560c06e0e0 legacy=0x7f560c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 10 monclient: shutdown 2024-03-29T11:25:59.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.038+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 >> v1:172.21.15.117:6790/0 conn(0x7f562c137e50 legacy=0x7f562c1e5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.039+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 shutdown_connections 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.039+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 >> 172.21.15.117:0/1086880080 conn(0x7f562c06b660 msgr2=0x7f562c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.039+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 shutdown_connections 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.039+0000 7f563114d640 1 -- 172.21.15.117:0/1086880080 wait complete. 2024-03-29T11:25:59.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:25:59.039+0000 7f563114d640 1 librados: shutdown 2024-03-29T11:25:59.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:05.844 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.002+0000 7f246e9f6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.002+0000 7f246e9f6640 10 monclient: build_initial_monmap 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.002+0000 7f246e9f6640 10 monclient: monmap: 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:05.989 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:06.003296+0000 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:06.003296+0000 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.003+0000 7f246e9f6640 1 Processor -- start 2024-03-29T11:26:05.990 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.003+0000 7f246e9f6640 1 -- start start 2024-03-29T11:26:05.991 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.003+0000 7f246e9f6640 10 monclient: init 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f24680af360 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2468150e60 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f24681541a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:06.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24680ab4b0 con 0x7f24681541a0 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24681545d0 con 0x7f2468150e60 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24680ac450 con 0x7f24680af360 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.004+0000 7f246e9f6640 10 monclient(hunting): authenticate will time out at 1234.313471s 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.007+0000 7f246cf93640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f24681541a0 0x7f246815b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:55685/0 (socket says 172.21.15.117:55685) 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.007+0000 7f246cf93640 1 -- 172.21.15.117:0/1905173502 learned_addr learned my addr 172.21.15.117:0/1905173502 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580578323 0 0) 0x7f24680ac450 con 0x7f24680af360 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2450005730 con 0x7f24680af360 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647704239 0 0) 0x7f24680ab4b0 con 0x7f24681541a0 2024-03-29T11:26:06.010 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.008+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24680ac450 con 0x7f24681541a0 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892537801 0 0) 0x7f2450005730 con 0x7f24680af360 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 >> v1:172.21.15.173:6789/0 conn(0x7f2468150e60 legacy=0x7f2468153a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 >> v1:172.21.15.117:6789/0 conn(0x7f24681541a0 legacy=0x7f246815b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 monclient: found mon.noname-b 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24680ab700 con 0x7f24680af360 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f24580015c0 con 0x7f24680af360 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: dump: 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f246e9f6640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f246e9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2458001720 con 0x7f24680af360 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 1 -- 172.21.15.117:0/1905173502 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2458001b30 con 0x7f24680af360 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f2466ffd640 10 monclient: dump: 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.013 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f246e9f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.009+0000 7f246e9f6640 4 set_mon_vals no callback set 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.010+0000 7f246e9f6640 10 monclient: shutdown 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.010+0000 7f246e9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.010+0000 7f246e9f6640 1 -- 172.21.15.117:0/1905173502 >> v1:172.21.15.117:6790/0 conn(0x7f24680af360 legacy=0x7f2468150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.011+0000 7f246e9f6640 1 -- 172.21.15.117:0/1905173502 shutdown_connections 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.011+0000 7f246e9f6640 1 -- 172.21.15.117:0/1905173502 >> 172.21.15.117:0/1905173502 conn(0x7f24680b35e0 msgr2=0x7f24680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.012+0000 7f246e9f6640 1 -- 172.21.15.117:0/1905173502 shutdown_connections 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.012+0000 7f246e9f6640 1 -- 172.21.15.117:0/1905173502 wait complete. 2024-03-29T11:26:06.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 10 monclient: build_initial_monmap 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 10 monclient: monmap: 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 1 librados: starting msgr at 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 1 librados: starting objecter 2024-03-29T11:26:06.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 1 Processor -- start 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.013+0000 7f246e9f6640 1 -- start start 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.014+0000 7f246e9f6640 1 librados: setting wanted keys 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.014+0000 7f246e9f6640 1 librados: calling monclient init 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.014+0000 7f246e9f6640 10 monclient: init 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.015+0000 7f246e9f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.015+0000 7f246e9f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.015+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f24680af360 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2468150e60 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f24681541a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246807c160 con 0x7f24680af360 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24681e74e0 con 0x7f2468150e60 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24681e6fc0 con 0x7f24681541a0 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:06.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.017+0000 7f246e9f6640 10 monclient(hunting): authenticate will time out at 1234.326803s 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f246cf93640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f24681541a0 0x7f24681ef300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47650/0 (socket says 172.21.15.117:47650) 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f246cf93640 1 -- 172.21.15.117:0/1083386682 learned_addr learned my addr 172.21.15.117:0/1083386682 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 client.?.objecter ms_handle_connect 0x7f24681541a0 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 client.?.objecter ms_handle_connect 0x7f2468150e60 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234882107 0 0) 0x7f24681e74e0 con 0x7f2468150e60 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.018+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454004eb0 con 0x7f2468150e60 2024-03-29T11:26:06.017 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.019+0000 7f2464ff9640 10 client.?.objecter ms_handle_connect 0x7f24680af360 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445291182 0 0) 0x7f2454004eb0 con 0x7f2468150e60 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24681e74e0 con 0x7f2468150e60 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f245c001870 con 0x7f2468150e60 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:06.018 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694336868 0 0) 0x7f246807c160 con 0x7f24680af360 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:06.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.020+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454004eb0 con 0x7f24680af360 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3982474525 0 0) 0x7f2454004eb0 con 0x7f24680af360 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f246807c160 con 0x7f24680af360 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2458001a30 con 0x7f24680af360 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.020 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 378928580 0 0) 0x7f24681e74e0 con 0x7f2468150e60 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 >> v1:172.21.15.117:6790/0 conn(0x7f24681541a0 legacy=0x7f24681ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 >> v1:172.21.15.117:6789/0 conn(0x7f24680af360 legacy=0x7f24681e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 monclient: found mon.b 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24681e7150 con 0x7f2468150e60 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:06.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f2464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 10 monclient: _renew_subs 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246808a820 con 0x7f2468150e60 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 10 monclient: _renew_subs 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f246814dfa0 con 0x7f2468150e60 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.021+0000 7f246e9f6640 1 librados: init done 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f245c001dc0 con 0x7f2468150e60 2024-03-29T11:26:06.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f245c003830 con 0x7f2468150e60 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.022+0000 7f2464ff9640 10 monclient: dump: 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:06.023 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.024+0000 7f246e9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.024+0000 7f246e9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.024+0000 7f246e9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:06.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.024+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24681e7340 con 0x7f2468150e60 2024-03-29T11:26:06.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.024+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f245c019800 con 0x7f2468150e60 2024-03-29T11:26:06.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f245c083870 con 0x7f2468150e60 2024-03-29T11:26:06.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2464ff9640 10 client.4459.objecter ms_dispatch 0x7f2468000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2464ff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2464ff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2464ff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f2467fff640 10 monclient: get_auth_request con 0x7f245406dee0 auth_method 0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.025+0000 7f24667fc640 4 set_mon_vals no callback set 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.026+0000 7f2464ff9640 10 client.4459.objecter ms_handle_connect 0x7f245406dee0 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.034+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24681e7340 con 0x7f2468150e60 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.034+0000 7f2464ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:06.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.034+0000 7f2464ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:06.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.358+0000 7f246e9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.358+0000 7f246e9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.358+0000 7f246e9f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:06.344 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.358+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24681e7670 con 0x7f2468150e60 2024-03-29T11:26:06.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.360+0000 7f2464ff9640 1 -- 172.21.15.117:0/1083386682 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f245c04f150 con 0x7f2468150e60 2024-03-29T11:26:06.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.360+0000 7f2464ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:06.345 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.360+0000 7f2464ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:06.345 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:06.345 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:06.348 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.363+0000 7f246e9f6640 10 librados: watch_flush enter 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.363+0000 7f246e9f6640 10 librados: watch_flush exit 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.363+0000 7f246e9f6640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.363+0000 7f246e9f6640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 >> v1:172.21.15.117:6816/4276038175 conn(0x7f245406dee0 legacy=0x7f24540703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 10 monclient: shutdown 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:06.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 >> v1:172.21.15.173:6789/0 conn(0x7f2468150e60 legacy=0x7f246814d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:06.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 shutdown_connections 2024-03-29T11:26:06.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.364+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 >> 172.21.15.117:0/1083386682 conn(0x7f24680b35e0 msgr2=0x7f24680a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:06.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.365+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 shutdown_connections 2024-03-29T11:26:06.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.366+0000 7f246e9f6640 1 -- 172.21.15.117:0/1083386682 wait complete. 2024-03-29T11:26:06.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:06.366+0000 7f246e9f6640 1 librados: shutdown 2024-03-29T11:26:06.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:13.178 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:13.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.341+0000 7f973aad4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:13.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.341+0000 7f973aad4640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.327 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.341+0000 7f973aad4640 10 monclient: monmap: 2024-03-29T11:26:13.327 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:13.327 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:13.342817+0000 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:13.342817+0000 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:13.328 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.345+0000 7f973aad4640 1 Processor -- start 2024-03-29T11:26:13.332 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.345+0000 7f973aad4640 1 -- start start 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.345+0000 7f973aad4640 10 monclient: init 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-b con 0x7f97340b1f60 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-a con 0x7f9734071eb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-c con 0x7f9734143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973406ef80 con 0x7f9734071eb0 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973406f260 con 0x7f9734143d10 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340b40c0 con 0x7f97340b1f60 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.333 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.346+0000 7f973aad4640 10 monclient(hunting): authenticate will time out at 1241.655835s 2024-03-29T11:26:13.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.349+0000 7f9738870640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f97340b1f60 0x7f9734143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59312/0 (socket says 172.21.15.117:59312) 2024-03-29T11:26:13.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.349+0000 7f9738870640 1 -- 172.21.15.117:0/1217194991 learned_addr learned my addr 172.21.15.117:0/1217194991 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:13.335 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910157847 0 0) 0x7f97340b40c0 con 0x7f97340b1f60 2024-03-29T11:26:13.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:26:13.336 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9724003520 con 0x7f97340b1f60 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 794681411 0 0) 0x7f9724003520 con 0x7f97340b1f60 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 >> v1:172.21.15.173:6789/0 conn(0x7f9734143d10 legacy=0x7f9734146b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 >> v1:172.21.15.117:6789/0 conn(0x7f9734071eb0 legacy=0x7f9734146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.337 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734155d70 con 0x7f97340b1f60 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9728001690 con 0x7f97340b1f60 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.350+0000 7f97337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: dump: 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.338 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f973aad4640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f973aad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9728001960 con 0x7f97340b1f60 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 1 -- 172.21.15.117:0/1217194991 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9728001cf0 con 0x7f97340b1f60 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f97337fe640 10 monclient: dump: 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.339 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f973aad4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:13.340 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.351+0000 7f973aad4640 4 set_mon_vals no callback set 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 10 monclient: shutdown 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 1 -- 172.21.15.117:0/1217194991 >> v1:172.21.15.117:6790/0 conn(0x7f97340b1f60 legacy=0x7f9734143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 1 -- 172.21.15.117:0/1217194991 shutdown_connections 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 1 -- 172.21.15.117:0/1217194991 >> 172.21.15.117:0/1217194991 conn(0x7f9734066a00 msgr2=0x7f9734066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.341 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 1 -- 172.21.15.117:0/1217194991 shutdown_connections 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.352+0000 7f973aad4640 1 -- 172.21.15.117:0/1217194991 wait complete. 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.353+0000 7f973aad4640 10 monclient: build_initial_monmap 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.353+0000 7f973aad4640 10 monclient: monmap: 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 librados: starting msgr at 2024-03-29T11:26:13.342 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 librados: starting objecter 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 Processor -- start 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 -- start start 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 librados: setting wanted keys 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 1 librados: calling monclient init 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.354+0000 7f973aad4640 10 monclient: init 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:13.343 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-a con 0x7f9734143d10 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-c con 0x7f9734140460 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 10 monclient(hunting): picked mon.noname-b con 0x7f97340ab560 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340ad110 con 0x7f9734143d10 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.355+0000 7f973aad4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340abad0 con 0x7f97340ab560 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f9733fff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9734140460 0x7f97341e3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:60620/0 (socket says 172.21.15.117:60620) 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f9733fff640 1 -- 172.21.15.117:0/2323404665 learned_addr learned my addr 172.21.15.117:0/2323404665 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 client.?.objecter ms_handle_connect 0x7f9734140460 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734146140 con 0x7f9734140460 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f973aad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f973aad4640 10 monclient(hunting): authenticate will time out at 1241.668804s 2024-03-29T11:26:13.349 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948584284 0 0) 0x7f9734146140 con 0x7f9734140460 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.359+0000 7f97317fa640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9718004eb0 con 0x7f9734140460 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 client.?.objecter ms_handle_connect 0x7f97340ab560 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103109987 0 0) 0x7f97340abad0 con 0x7f97340ab560 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:13.350 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734146140 con 0x7f97340ab560 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 568318827 0 0) 0x7f9718004eb0 con 0x7f9734140460 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9718005490 con 0x7f9734140460 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9720001a30 con 0x7f9734140460 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): dump: 2024-03-29T11:26:13.351 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847558559 0 0) 0x7f9734146140 con 0x7f97340ab560 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9718004eb0 con 0x7f97340ab560 2024-03-29T11:26:13.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f972c001a70 con 0x7f97340ab560 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 10 monclient(hunting): dump: 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668716777 0 0) 0x7f9718005490 con 0x7f9734140460 2024-03-29T11:26:13.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 >> v1:172.21.15.173:6789/0 conn(0x7f97340ab560 legacy=0x7f97341e46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.354 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.360+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 >> v1:172.21.15.117:6789/0 conn(0x7f9734143d10 legacy=0x7f973413fd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:13.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:13.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 1 monclient: found mon.c 2024-03-29T11:26:13.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:13.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97341462d0 con 0x7f9734140460 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f97317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 10 monclient: _renew_subs 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97340aae70 con 0x7f9734140460 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 10 monclient: _renew_subs 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341f0820 con 0x7f9734140460 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.364+0000 7f973aad4640 1 librados: init done 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f973aad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f973aad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f973aad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97341464c0 con 0x7f9734140460 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9720001c70 con 0x7f9734140460 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9720003a60 con 0x7f9734140460 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 10 monclient: dump: 2024-03-29T11:26:13.359 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.367+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f9720003c40 con 0x7f9734140460 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97327fc640 4 set_mon_vals no callback set 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f9720084ae0 con 0x7f9734140460 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97317fa640 10 client.4550.objecter ms_dispatch 0x7f9734000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97317fa640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:13.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97317fa640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f97317fa640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.368+0000 7f9738870640 10 monclient: get_auth_request con 0x7f971806e180 auth_method 0 2024-03-29T11:26:13.361 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.369+0000 7f97317fa640 10 client.4550.objecter ms_handle_connect 0x7f971806e180 2024-03-29T11:26:13.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.377+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9720050520 con 0x7f9734140460 2024-03-29T11:26:13.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.377+0000 7f97317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:13.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.377+0000 7f97317fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:13.690 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.705+0000 7f973aad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.705+0000 7f973aad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.705+0000 7f973aad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:13.691 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.705+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9734075f10 con 0x7f9734140460 2024-03-29T11:26:13.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.710+0000 7f97317fa640 1 -- 172.21.15.117:0/2323404665 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f9720052450 con 0x7f9734140460 2024-03-29T11:26:13.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.710+0000 7f97317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:13.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.710+0000 7f97317fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:13.698 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:13.699 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:13.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 10 librados: watch_flush enter 2024-03-29T11:26:13.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 10 librados: watch_flush exit 2024-03-29T11:26:13.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:26:13.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 >> v1:172.21.15.117:6816/4276038175 conn(0x7f971806e180 legacy=0x7f9718070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 10 monclient: shutdown 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.713+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 >> v1:172.21.15.117:6790/0 conn(0x7f9734140460 legacy=0x7f97341e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.714+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 shutdown_connections 2024-03-29T11:26:13.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.714+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 >> 172.21.15.117:0/2323404665 conn(0x7f9734066a00 msgr2=0x7f97340b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:13.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.715+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 shutdown_connections 2024-03-29T11:26:13.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.715+0000 7f973aad4640 1 -- 172.21.15.117:0/2323404665 wait complete. 2024-03-29T11:26:13.701 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:13.715+0000 7f973aad4640 1 librados: shutdown 2024-03-29T11:26:13.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:20.524 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:20.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.694+0000 7f99410c1640 10 monclient: get_monmap_and_config 2024-03-29T11:26:20.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.694+0000 7f99410c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.680 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.694+0000 7f99410c1640 10 monclient: monmap: 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:20.695690+0000 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:20.695690+0000 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.681 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.695+0000 7f99410c1640 1 Processor -- start 2024-03-29T11:26:20.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.695+0000 7f99410c1640 1 -- start start 2024-03-29T11:26:20.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.695+0000 7f99410c1640 10 monclient: init 2024-03-29T11:26:20.682 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.696+0000 7f99410c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.685 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.696+0000 7f99410c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:20.686 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.696+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-a con 0x7f9934005ff0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:20.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.696+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9934033640 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:20.692 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.696+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-b con 0x7f9934031ee0 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99410c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340ccd50 con 0x7f9934005ff0 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99410c1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9934032980 con 0x7f9934033640 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99410c1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9934032c60 con 0x7f9934031ee0 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99410c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99410c1640 10 monclient(hunting): authenticate will time out at 1249.006039s 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99408c0640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9934031ee0 0x7f99340cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:43438/0 (socket says 172.21.15.117:43438) 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f99408c0640 1 -- 172.21.15.117:0/1647639891 learned_addr learned my addr 172.21.15.117:0/1647639891 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474122932 0 0) 0x7f9934032c60 con 0x7f9934031ee0 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:26:20.693 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992c003520 con 0x7f9934031ee0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139063004 0 0) 0x7f992c003520 con 0x7f9934031ee0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.697+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 >> v1:172.21.15.173:6789/0 conn(0x7f9934033640 legacy=0x7f99340cbf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.700+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 >> v1:172.21.15.117:6789/0 conn(0x7f9934005ff0 legacy=0x7f9934031500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993bfff640 1 -- 172.21.15.117:0/1647639891 >> v1:172.21.15.117:6789/0 conn(0x7f9934005ff0 legacy=0x7f9934031500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 1 monclient: found mon.noname-b 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9934032f40 con 0x7f9934031ee0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.694 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f993c04f730 con 0x7f9934031ee0 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: dump: 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:20.695 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f99410c1640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f99410c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f993c0008d0 con 0x7f9934031ee0 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 1 -- 172.21.15.117:0/1647639891 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f993c0758b0 con 0x7f9934031ee0 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f993affd640 10 monclient: dump: 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:20.696 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f99410c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.701+0000 7f99410c1640 4 set_mon_vals no callback set 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.702+0000 7f99410c1640 10 monclient: shutdown 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.702+0000 7f99410c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.702+0000 7f99410c1640 1 -- 172.21.15.117:0/1647639891 >> v1:172.21.15.117:6790/0 conn(0x7f9934031ee0 legacy=0x7f99340cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.703+0000 7f99410c1640 1 -- 172.21.15.117:0/1647639891 shutdown_connections 2024-03-29T11:26:20.697 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.703+0000 7f99410c1640 1 -- 172.21.15.117:0/1647639891 >> 172.21.15.117:0/1647639891 conn(0x7f99340065b0 msgr2=0x7f9934006990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.703+0000 7f99410c1640 1 -- 172.21.15.117:0/1647639891 shutdown_connections 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.703+0000 7f99410c1640 1 -- 172.21.15.117:0/1647639891 wait complete. 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.704+0000 7f99410c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.704+0000 7f99410c1640 10 monclient: monmap: 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:20.698 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 librados: starting msgr at 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 librados: starting objecter 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 Processor -- start 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 -- start start 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 librados: setting wanted keys 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 1 librados: calling monclient init 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.705+0000 7f99410c1640 10 monclient: init 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-b con 0x7f9934005ff0 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-a con 0x7f9934031ee0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:20.699 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9934033640 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:20.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993402e2c0 con 0x7f9934031ee0 2024-03-29T11:26:20.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993402e5a0 con 0x7f9934005ff0 2024-03-29T11:26:20.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993402e880 con 0x7f9934033640 2024-03-29T11:26:20.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:20.700 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.706+0000 7f99410c1640 10 monclient(hunting): authenticate will time out at 1249.015315s 2024-03-29T11:26:20.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.708+0000 7f99408c0640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f9934033640 0x7f993402f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:45018/0 (socket says 172.21.15.117:45018) 2024-03-29T11:26:20.706 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.708+0000 7f99408c0640 1 -- 172.21.15.117:0/2982394979 learned_addr learned my addr 172.21.15.117:0/2982394979 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 client.?.objecter ms_handle_connect 0x7f9934033640 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443486315 0 0) 0x7f993402e880 con 0x7f9934033640 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.709+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920004eb0 con 0x7f9934033640 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 client.?.objecter ms_handle_connect 0x7f9934005ff0 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3858823981 0 0) 0x7f9920004eb0 con 0x7f9934033640 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f993402e880 con 0x7f9934033640 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 client.?.objecter ms_handle_connect 0x7f9934031ee0 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:20.707 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f993c04f400 con 0x7f9934033640 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.708 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235807106 0 0) 0x7f993402e5a0 con 0x7f9934005ff0 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920004eb0 con 0x7f9934005ff0 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304576569 0 0) 0x7f993402e2c0 con 0x7f9934031ee0 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:20.709 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.710+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993402e5a0 con 0x7f9934031ee0 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962537679 0 0) 0x7f993402e880 con 0x7f9934033640 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 >> v1:172.21.15.173:6789/0 conn(0x7f9934005ff0 legacy=0x7f9934029610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 >> v1:172.21.15.117:6789/0 conn(0x7f9934031ee0 legacy=0x7f993402eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 monclient: found mon.c 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993402eb60 con 0x7f9934033640 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f993c04f730 con 0x7f9934033640 2024-03-29T11:26:20.710 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f993c067d90 con 0x7f9934033640 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f9938ff9640 10 monclient: dump: 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:20.711 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.711+0000 7f993a7fc640 4 set_mon_vals no callback set 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 10 monclient: _renew_subs 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993402fe10 con 0x7f9934033640 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 10 monclient: _renew_subs 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f993402c240 con 0x7f9934033640 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.712+0000 7f99410c1640 1 librados: init done 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.714+0000 7f99410c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.714+0000 7f99410c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.714+0000 7f99410c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.714+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993402d200 con 0x7f9934033640 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.714+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f993c067ef0 con 0x7f9934033640 2024-03-29T11:26:20.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.715+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f993c0ff680 con 0x7f9934033640 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.715+0000 7f9938ff9640 10 client.4556.objecter ms_dispatch 0x7f993c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.715+0000 7f9938ff9640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.715+0000 7f9938ff9640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.715+0000 7f9938ff9640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.716+0000 7f993bfff640 10 monclient: get_auth_request con 0x7f992006dc30 auth_method 0 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.716+0000 7f9938ff9640 10 client.4556.objecter ms_handle_connect 0x7f992006dc30 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.723+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f993c0caf60 con 0x7f9934033640 2024-03-29T11:26:20.713 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.723+0000 7f9938ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:20.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:20.723+0000 7f9938ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:21.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.046+0000 7f99410c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.047+0000 7f99410c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.047+0000 7f99410c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:21.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.047+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f993402c970 con 0x7f9934033640 2024-03-29T11:26:21.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.050+0000 7f9938ff9640 1 -- 172.21.15.117:0/2982394979 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f993c0cce90 con 0x7f9934033640 2024-03-29T11:26:21.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.050+0000 7f9938ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:21.036 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.050+0000 7f9938ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:21.036 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:21.036 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 10 librados: watch_flush enter 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 10 librados: watch_flush exit 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 >> v1:172.21.15.117:6816/4276038175 conn(0x7f992006dc30 legacy=0x7f99200700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 10 monclient: shutdown 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.053+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 >> v1:172.21.15.117:6790/0 conn(0x7f9934033640 legacy=0x7f993402f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.054+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 shutdown_connections 2024-03-29T11:26:21.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.054+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 >> 172.21.15.117:0/2982394979 conn(0x7f99340065b0 msgr2=0x7f99340043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:21.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.055+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 shutdown_connections 2024-03-29T11:26:21.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.055+0000 7f99410c1640 1 -- 172.21.15.117:0/2982394979 wait complete. 2024-03-29T11:26:21.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:21.055+0000 7f99410c1640 1 librados: shutdown 2024-03-29T11:26:21.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:27.867 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.035+0000 7f2729fcf640 10 monclient: get_monmap_and_config 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.035+0000 7f2729fcf640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.035+0000 7f2729fcf640 10 monclient: monmap: 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:28.036904+0000 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:28.036904+0000 2024-03-29T11:26:28.022 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.037+0000 7f2729fcf640 1 Processor -- start 2024-03-29T11:26:28.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.037+0000 7f2729fcf640 1 -- start start 2024-03-29T11:26:28.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.037+0000 7f2729fcf640 10 monclient: init 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f27241457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f272413f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f27240a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27240aaef0 con 0x7f27240a9d70 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724145bd0 con 0x7f272413f650 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724145eb0 con 0x7f27241457f0 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2722ffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f272413f650 0x7f27240b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:47324/0 (socket says 172.21.15.117:47324) 2024-03-29T11:26:28.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2723fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f27240a9d70 0x7f272414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:38862/0 (socket says 172.21.15.117:38862) 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2723fff640 1 -- 172.21.15.117:0/3490140685 learned_addr learned my addr 172.21.15.117:0/3490140685 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.038+0000 7f2729fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f2729fcf640 10 monclient(hunting): authenticate will time out at 1256.348151s 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945368599 0 0) 0x7f2724145eb0 con 0x7f27241457f0 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c003520 con 0x7f27241457f0 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607821018 0 0) 0x7f27240aaef0 con 0x7f27240a9d70 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.039+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2724145eb0 con 0x7f27240a9d70 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604079502 0 0) 0x7f2724145eb0 con 0x7f27240a9d70 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 >> v1:172.21.15.117:6790/0 conn(0x7f27241457f0 legacy=0x7f272413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 >> v1:172.21.15.173:6789/0 conn(0x7f272413f650 legacy=0x7f27240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724146340 con 0x7f27240a9d70 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f271c001970 con 0x7f27240a9d70 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f2729fcf640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:26:28.031 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f2729fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f27227fc640 10 monclient: dump: 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.032 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.040+0000 7f2729fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f271c001b90 con 0x7f27240a9d70 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 1 -- 172.21.15.117:0/3490140685 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f271c001f20 con 0x7f27240a9d70 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f27227fc640 10 monclient: dump: 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:28.033 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f2729fcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.041+0000 7f2729fcf640 4 set_mon_vals no callback set 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 10 monclient: shutdown 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 1 -- 172.21.15.117:0/3490140685 >> v1:172.21.15.117:6789/0 conn(0x7f27240a9d70 legacy=0x7f272414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 1 -- 172.21.15.117:0/3490140685 shutdown_connections 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 1 -- 172.21.15.117:0/3490140685 >> 172.21.15.117:0/3490140685 conn(0x7f2724066a00 msgr2=0x7f2724066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 1 -- 172.21.15.117:0/3490140685 shutdown_connections 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.042+0000 7f2729fcf640 1 -- 172.21.15.117:0/3490140685 wait complete. 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.043+0000 7f2729fcf640 10 monclient: build_initial_monmap 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.043+0000 7f2729fcf640 10 monclient: monmap: 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:28.038 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 librados: starting msgr at 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 librados: starting objecter 2024-03-29T11:26:28.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 Processor -- start 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 -- start start 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 librados: setting wanted keys 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 1 librados: calling monclient init 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.044+0000 7f2729fcf640 10 monclient: init 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f27240a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f272413f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2722ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f272413f650 0x7f27241421f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44046/0 (socket says 172.21.15.117:44046) 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f27241457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2722ffd640 1 -- 172.21.15.117:0/1659999897 learned_addr learned my addr 172.21.15.117:0/1659999897 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27241401a0 con 0x7f27240a9d70 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f270bfff640 10 client.?.objecter ms_handle_connect 0x7f272413f650 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f270bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27241db540 con 0x7f27241457f0 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2724142900 con 0x7f272413f650 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f2729fcf640 10 monclient(hunting): authenticate will time out at 1256.354783s 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f270bfff640 10 client.?.objecter ms_handle_connect 0x7f27240a9d70 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.045+0000 7f270bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891371176 0 0) 0x7f27241401a0 con 0x7f27240a9d70 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710004eb0 con 0x7f27240a9d70 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853547951 0 0) 0x7f2724142900 con 0x7f272413f650 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27241401a0 con 0x7f272413f650 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1670875559 0 0) 0x7f27241401a0 con 0x7f272413f650 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2710005490 con 0x7f272413f650 2024-03-29T11:26:28.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524786620 0 0) 0x7f2710004eb0 con 0x7f27240a9d70 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2710005a70 con 0x7f27240a9d70 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2714001800 con 0x7f27240a9d70 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.046+0000 7f270bfff640 10 monclient(hunting): dump: 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:28.043 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2718001950 con 0x7f272413f650 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): dump: 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.044 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 client.?.objecter ms_handle_connect 0x7f27241457f0 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440071535 0 0) 0x7f27241db540 con 0x7f27241457f0 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710004eb0 con 0x7f27241457f0 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167349758 0 0) 0x7f2710005490 con 0x7f272413f650 2024-03-29T11:26:28.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 >> v1:172.21.15.173:6789/0 conn(0x7f27241457f0 legacy=0x7f27241d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 >> v1:172.21.15.117:6789/0 conn(0x7f27240a9d70 legacy=0x7f27241da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 monclient: found mon.c 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724141dd0 con 0x7f272413f650 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f270bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 10 monclient: _renew_subs 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:28.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272413fda0 con 0x7f272413f650 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 10 monclient: _renew_subs 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.047+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f272413b400 con 0x7f272413f650 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f2729fcf640 1 librados: init done 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27180047e0 con 0x7f272413f650 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2718004af0 con 0x7f272413f650 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f270bfff640 10 monclient: dump: 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.047 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.048+0000 7f27217fa640 4 set_mon_vals no callback set 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.049+0000 7f2729fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.049+0000 7f2729fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.049+0000 7f2729fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.049+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704005080 con 0x7f272413f650 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.050+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f271801aac0 con 0x7f272413f650 2024-03-29T11:26:28.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f2718084a20 con 0x7f272413f650 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 10 client.4562.objecter ms_dispatch 0x7f2724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f27237fe640 10 monclient: get_auth_request con 0x7f271006e490 auth_method 0 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.051+0000 7f270bfff640 10 client.4562.objecter ms_handle_connect 0x7f271006e490 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.060+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2718050130 con 0x7f272413f650 2024-03-29T11:26:28.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.060+0000 7f270bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:28.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.060+0000 7f270bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.378+0000 7f2729fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.378+0000 7f2729fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.378+0000 7f2729fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.378+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2704005330 con 0x7f272413f650 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.379+0000 7f270bfff640 1 -- 172.21.15.117:0/1659999897 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f2718052230 con 0x7f272413f650 2024-03-29T11:26:28.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.379+0000 7f270bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:28.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.379+0000 7f270bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:28.366 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:28.366 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:28.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.382+0000 7f2729fcf640 10 librados: watch_flush enter 2024-03-29T11:26:28.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 10 librados: watch_flush exit 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 >> v1:172.21.15.117:6816/4276038175 conn(0x7f271006e490 legacy=0x7f2710070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 10 monclient: shutdown 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:28.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.383+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 >> v1:172.21.15.117:6790/0 conn(0x7f272413f650 legacy=0x7f27241421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:28.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.386+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 shutdown_connections 2024-03-29T11:26:28.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.386+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 >> 172.21.15.117:0/1659999897 conn(0x7f2724066a00 msgr2=0x7f27240b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:28.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.386+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 shutdown_connections 2024-03-29T11:26:28.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.386+0000 7f2729fcf640 1 -- 172.21.15.117:0/1659999897 wait complete. 2024-03-29T11:26:28.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:28.386+0000 7f2729fcf640 1 librados: shutdown 2024-03-29T11:26:28.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:35.192 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:35.352 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.366+0000 7fd6710d3640 10 monclient: get_monmap_and_config 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.366+0000 7fd6710d3640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.366+0000 7fd6710d3640 10 monclient: monmap: 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:35.367346+0000 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:35.367346+0000 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:35.353 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.367+0000 7fd6710d3640 1 Processor -- start 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.367+0000 7fd6710d3640 1 -- start start 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.367+0000 7fd6710d3640 10 monclient: init 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd66c06ab70 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd66c06af50 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd66c141fb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0b0d50 con 0x7fd66c141fb0 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0b1030 con 0x7fd66c06af50 2024-03-29T11:26:35.355 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0afa70 con 0x7fd66c06ab70 2024-03-29T11:26:35.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd6710d3640 10 monclient(hunting): authenticate will time out at 1263.677349s 2024-03-29T11:26:35.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd66b59e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd66c141fb0 0x7fd66c0715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:32906/0 (socket says 172.21.15.117:32906) 2024-03-29T11:26:35.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.368+0000 7fd66b59e640 1 -- 172.21.15.117:0/2370067353 learned_addr learned my addr 172.21.15.117:0/2370067353 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:35.356 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833654053 0 0) 0x7fd66c0b1030 con 0x7fd66c06af50 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd654003520 con 0x7fd66c06af50 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870301448 0 0) 0x7fd66c0b0d50 con 0x7fd66c141fb0 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd66c0b1030 con 0x7fd66c141fb0 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051543642 0 0) 0x7fd654003520 con 0x7fd66c06af50 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 >> v1:172.21.15.117:6790/0 conn(0x7fd66c06ab70 legacy=0x7fd66c0af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.357 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 >> v1:172.21.15.117:6789/0 conn(0x7fd66c141fb0 legacy=0x7fd66c0715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 monclient: found mon.noname-c 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c1423e0 con 0x7fd66c06af50 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd6710d3640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd65c001a10 con 0x7fd66c06af50 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd6710d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.369+0000 7fd669d9b640 10 monclient: dump: 2024-03-29T11:26:35.358 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd6710d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd65c001b70 con 0x7fd66c06af50 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd6710d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd6710d3640 4 set_mon_vals no callback set 2024-03-29T11:26:35.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 1 -- 172.21.15.117:0/2370067353 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd65c001f80 con 0x7fd66c06af50 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.370+0000 7fd669d9b640 10 monclient: dump: 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:35.360 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 10 monclient: shutdown 2024-03-29T11:26:35.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 1 -- 172.21.15.117:0/2370067353 >> v1:172.21.15.173:6789/0 conn(0x7fd66c06af50 legacy=0x7fd66c072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 1 -- 172.21.15.117:0/2370067353 shutdown_connections 2024-03-29T11:26:35.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 1 -- 172.21.15.117:0/2370067353 >> 172.21.15.117:0/2370067353 conn(0x7fd66c067470 msgr2=0x7fd66c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 1 -- 172.21.15.117:0/2370067353 shutdown_connections 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.371+0000 7fd6710d3640 1 -- 172.21.15.117:0/2370067353 wait complete. 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.372+0000 7fd6710d3640 10 monclient: build_initial_monmap 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.372+0000 7fd6710d3640 10 monclient: monmap: 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 librados: starting msgr at 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 librados: starting objecter 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 Processor -- start 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 -- start start 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 librados: setting wanted keys 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 1 librados: calling monclient init 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.373+0000 7fd6710d3640 10 monclient: init 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd66c06ab70 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd66c141fb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd66c139860 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c1deb30 con 0x7fd66c141fb0 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c1df760 con 0x7fd66c06ab70 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c1df8f0 con 0x7fd66c139860 2024-03-29T11:26:35.364 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6710d3640 10 monclient(hunting): authenticate will time out at 1263.683527s 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd66ad9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd66c06ab70 0x7fd66c0ae7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:40358/0 (socket says 172.21.15.117:40358) 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd66ad9d640 1 -- 172.21.15.117:0/1741416183 learned_addr learned my addr 172.21.15.117:0/1741416183 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6537fe640 10 client.?.objecter ms_handle_connect 0x7fd66c06ab70 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.374+0000 7fd6537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 client.?.objecter ms_handle_connect 0x7fd66c139860 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052196396 0 0) 0x7fd66c1df760 con 0x7fd66c06ab70 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd658004eb0 con 0x7fd66c06ab70 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 client.?.objecter ms_handle_connect 0x7fd66c141fb0 2024-03-29T11:26:35.365 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190949469 0 0) 0x7fd66c1df8f0 con 0x7fd66c139860 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.375+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd66c1df760 con 0x7fd66c139860 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417439093 0 0) 0x7fd658004eb0 con 0x7fd66c06ab70 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd66c1df8f0 con 0x7fd66c06ab70 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd664001a30 con 0x7fd66c06ab70 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): dump: 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.366 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 111254990 0 0) 0x7fd66c1df760 con 0x7fd66c139860 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd658005940 con 0x7fd66c139860 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd660001810 con 0x7fd66c139860 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): dump: 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:35.367 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:35.368 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.369 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861010871 0 0) 0x7fd66c1deb30 con 0x7fd66c141fb0 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd66c1df760 con 0x7fd66c141fb0 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1957649557 0 0) 0x7fd66c1df8f0 con 0x7fd66c06ab70 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 >> v1:172.21.15.117:6790/0 conn(0x7fd66c139860 legacy=0x7fd66c138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 >> v1:172.21.15.117:6789/0 conn(0x7fd66c141fb0 legacy=0x7fd66c1df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 monclient: found mon.b 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0ab120 con 0x7fd66c06ab70 2024-03-29T11:26:35.370 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 10 monclient: _renew_subs 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c0ab310 con 0x7fd66c06ab70 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 10 monclient: _renew_subs 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0adce0 con 0x7fd66c06ab70 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.376+0000 7fd6710d3640 1 librados: init done 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd664001c70 con 0x7fd66c06ab70 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd66959a640 4 set_mon_vals no callback set 2024-03-29T11:26:35.371 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd664001a30 con 0x7fd66c06ab70 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.377+0000 7fd6537fe640 10 monclient: dump: 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.378+0000 7fd6710d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.372 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.378+0000 7fd6710d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.378+0000 7fd6710d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.378+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c005080 con 0x7fd66c06ab70 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd6640038c0 con 0x7fd66c06ab70 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fd6640847b0 con 0x7fd66c06ab70 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 10 client.4477.objecter ms_dispatch 0x7fd66c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd6537fe640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.379+0000 7fd66a59c640 10 monclient: get_auth_request con 0x7fd65806e390 auth_method 0 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.380+0000 7fd6537fe640 10 client.4477.objecter ms_handle_connect 0x7fd65806e390 2024-03-29T11:26:35.373 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.386+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd664050090 con 0x7fd66c06ab70 2024-03-29T11:26:35.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.386+0000 7fd6537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:35.374 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.386+0000 7fd6537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:35.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.752+0000 7fd6710d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.752+0000 7fd6710d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.752+0000 7fd6710d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:35.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.752+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd64c005330 con 0x7fd66c06ab70 2024-03-29T11:26:35.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.753+0000 7fd6537fe640 1 -- 172.21.15.117:0/1741416183 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fd664051fc0 con 0x7fd66c06ab70 2024-03-29T11:26:35.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.753+0000 7fd6537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:35.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.753+0000 7fd6537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:35.740 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:35.740 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:35.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.756+0000 7fd6710d3640 10 librados: watch_flush enter 2024-03-29T11:26:35.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 10 librados: watch_flush exit 2024-03-29T11:26:35.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:26:35.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:26:35.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd65806e390 legacy=0x7fd658070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 10 monclient: shutdown 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 >> v1:172.21.15.173:6789/0 conn(0x7fd66c06ab70 legacy=0x7fd66c0ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 shutdown_connections 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 >> 172.21.15.117:0/1741416183 conn(0x7fd66c067470 msgr2=0x7fd66c0af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 shutdown_connections 2024-03-29T11:26:35.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 -- 172.21.15.117:0/1741416183 wait complete. 2024-03-29T11:26:35.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:35.757+0000 7fd6710d3640 1 librados: shutdown 2024-03-29T11:26:35.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:42.566 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.725+0000 7fa45e694640 10 monclient: get_monmap_and_config 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.725+0000 7fa45e694640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.725+0000 7fa45e694640 10 monclient: monmap: 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:42.726487+0000 2024-03-29T11:26:42.712 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:42.726487+0000 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:42.713 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.714 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.728+0000 7fa45e694640 1 Processor -- start 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.728+0000 7fa45e694640 1 -- start start 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.728+0000 7fa45e694640 10 monclient: init 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.729+0000 7fa45e694640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.729+0000 7fa45e694640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.729+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-b con 0x7fa4580aac30 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.729+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-a con 0x7fa458072f20 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.729+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-c con 0x7fa4580a9e90 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45e694640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580ad8e0 con 0x7fa458072f20 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45e694640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580ac6a0 con 0x7fa4580a9e90 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45e694640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580ab230 con 0x7fa4580aac30 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45e694640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45e694640 10 monclient(hunting): authenticate will time out at 1271.039255s 2024-03-29T11:26:42.717 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45d692640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa4580aac30 0x7fa45813af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47922/0 (socket says 172.21.15.117:47922) 2024-03-29T11:26:42.718 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.730+0000 7fa45d692640 1 -- 172.21.15.117:0/3664082176 learned_addr learned my addr 172.21.15.117:0/3664082176 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613045005 0 0) 0x7fa4580ab230 con 0x7fa4580aac30 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:26:42.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.732+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa444003520 con 0x7fa4580aac30 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.733+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930135575 0 0) 0x7fa444003520 con 0x7fa4580aac30 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.733+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 >> v1:172.21.15.173:6789/0 conn(0x7fa4580a9e90 legacy=0x7fa4580ad1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 >> v1:172.21.15.117:6789/0 conn(0x7fa458072f20 legacy=0x7fa4580acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 1 monclient: found mon.noname-b 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45813b660 con 0x7fa4580aac30 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.721 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.734+0000 7fa44ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa454001690 con 0x7fa4580aac30 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa44ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa44ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa44ffff640 10 monclient: dump: 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:42.722 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa45e694640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:26:42.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.735+0000 7fa45e694640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa454001920 con 0x7fa4580aac30 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 1 -- 172.21.15.117:0/3664082176 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa454001cf0 con 0x7fa4580aac30 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa44ffff640 10 monclient: dump: 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:42.725 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa45e694640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:42.726 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.738+0000 7fa45e694640 4 set_mon_vals no callback set 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 10 monclient: shutdown 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 1 -- 172.21.15.117:0/3664082176 >> v1:172.21.15.117:6790/0 conn(0x7fa4580aac30 legacy=0x7fa45813af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 1 -- 172.21.15.117:0/3664082176 shutdown_connections 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 1 -- 172.21.15.117:0/3664082176 >> 172.21.15.117:0/3664082176 conn(0x7fa4580b3a30 msgr2=0x7fa4580b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 1 -- 172.21.15.117:0/3664082176 shutdown_connections 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.739+0000 7fa45e694640 1 -- 172.21.15.117:0/3664082176 wait complete. 2024-03-29T11:26:42.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.740+0000 7fa45e694640 10 monclient: build_initial_monmap 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.740+0000 7fa45e694640 10 monclient: monmap: 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 librados: starting msgr at 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 librados: starting objecter 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 Processor -- start 2024-03-29T11:26:42.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 -- start start 2024-03-29T11:26:42.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 librados: setting wanted keys 2024-03-29T11:26:42.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.741+0000 7fa45e694640 1 librados: calling monclient init 2024-03-29T11:26:42.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.744+0000 7fa45e694640 10 monclient: init 2024-03-29T11:26:42.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.744+0000 7fa45e694640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:42.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.745+0000 7fa45e694640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:42.734 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.745+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-b con 0x7fa458072f20 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:42.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.747+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-c con 0x7fa4580a9e90 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45d692640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fa458072f20 0x7fa4581d6cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:45655/0 (socket says 172.21.15.117:45655) 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 10 monclient(hunting): picked mon.noname-a con 0x7fa4580aac30 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa458131370 con 0x7fa4580aac30 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4581374c0 con 0x7fa458072f20 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4581348d0 con 0x7fa4580a9e90 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:42.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45e694640 10 monclient(hunting): authenticate will time out at 1271.057258s 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa45d692640 1 -- 172.21.15.117:0/216285684 learned_addr learned my addr 172.21.15.117:0/216285684 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 client.?.objecter ms_handle_connect 0x7fa458072f20 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 client.?.objecter ms_handle_connect 0x7fa4580a9e90 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113994848 0 0) 0x7fa4581374c0 con 0x7fa458072f20 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.740 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.748+0000 7fa44dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43c004eb0 con 0x7fa458072f20 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948002552 0 0) 0x7fa4581348d0 con 0x7fa4580a9e90 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:42.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:26:42.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.749+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4581374c0 con 0x7fa4580a9e90 2024-03-29T11:26:42.748 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748843341 0 0) 0x7fa43c004eb0 con 0x7fa458072f20 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4581348d0 con 0x7fa458072f20 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3297628601 0 0) 0x7fa4581374c0 con 0x7fa4580a9e90 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa43c004eb0 con 0x7fa4580a9e90 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa454001cf0 con 0x7fa458072f20 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.749 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa448001870 con 0x7fa4580a9e90 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.752+0000 7fa44dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.750 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 390876051 0 0) 0x7fa4581348d0 con 0x7fa458072f20 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 >> v1:172.21.15.117:6790/0 conn(0x7fa4580a9e90 legacy=0x7fa458130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 >> v1:172.21.15.117:6789/0 conn(0x7fa4580aac30 legacy=0x7fa458130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 monclient: found mon.b 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458134a60 con 0x7fa458072f20 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:42.751 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa454004a90 con 0x7fa458072f20 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.753+0000 7fa44f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa45e694640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa45e694640 10 monclient: _renew_subs 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa45e694640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa458135d20 con 0x7fa458072f20 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa454004da0 con 0x7fa458072f20 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa44dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa44dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.754+0000 7fa44dffb640 10 monclient: dump: 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:42.752 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.755+0000 7fa45e694640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.755+0000 7fa45e694640 10 monclient: _renew_subs 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.755+0000 7fa45e694640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:42.753 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.755+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581d75a0 con 0x7fa458072f20 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.755+0000 7fa45e694640 1 librados: init done 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.757+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fa45401ad70 con 0x7fa458072f20 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fa454084b90 con 0x7fa458072f20 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa44dffb640 10 client.4480.objecter ms_dispatch 0x7fa458000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa44dffb640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:42.754 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa44dffb640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa44dffb640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.758+0000 7fa45ce91640 10 monclient: get_auth_request con 0x7fa43c06de90 auth_method 0 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.759+0000 7fa44dffb640 10 client.4480.objecter ms_handle_connect 0x7fa43c06de90 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.759+0000 7fa45e694640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.759+0000 7fa45e694640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.759+0000 7fa45e694640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:42.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.759+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa458135eb0 con 0x7fa458072f20 2024-03-29T11:26:42.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.766+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4540502a0 con 0x7fa458072f20 2024-03-29T11:26:42.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.766+0000 7fa44dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:42.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:42.766+0000 7fa44dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:43.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.135+0000 7fa45e694640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.135+0000 7fa45e694640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.135+0000 7fa45e694640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:26:43.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.135+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa45804f730 con 0x7fa458072f20 2024-03-29T11:26:43.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.136+0000 7fa44dffb640 1 -- 172.21.15.117:0/216285684 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fa4540523a0 con 0x7fa458072f20 2024-03-29T11:26:43.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.136+0000 7fa44dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:43.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.136+0000 7fa44dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:43.123 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:43.123 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:43.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.140+0000 7fa45e694640 10 librados: watch_flush enter 2024-03-29T11:26:43.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.141+0000 7fa45e694640 10 librados: watch_flush exit 2024-03-29T11:26:43.128 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.141+0000 7fa45e694640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:26:43.129 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.141+0000 7fa45e694640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:26:43.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.142+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 >> v1:172.21.15.117:6816/4276038175 conn(0x7fa43c06de90 legacy=0x7fa43c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.130 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.142+0000 7fa45e694640 10 monclient: shutdown 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.142+0000 7fa45e694640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.142+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 >> v1:172.21.15.173:6789/0 conn(0x7fa458072f20 legacy=0x7fa4581d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.143+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 shutdown_connections 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.143+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 >> 172.21.15.117:0/216285684 conn(0x7fa4580b3a30 msgr2=0x7fa4580aee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.143+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 shutdown_connections 2024-03-29T11:26:43.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.143+0000 7fa45e694640 1 -- 172.21.15.117:0/216285684 wait complete. 2024-03-29T11:26:43.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:43.143+0000 7fa45e694640 1 librados: shutdown 2024-03-29T11:26:43.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:49.958 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:50.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.125+0000 7f4336464640 10 monclient: get_monmap_and_config 2024-03-29T11:26:50.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.125+0000 7f4336464640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.125+0000 7f4336464640 10 monclient: monmap: 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:50.126546+0000 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:50.126546+0000 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:50.113 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.126+0000 7f4336464640 1 Processor -- start 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.126+0000 7f4336464640 1 -- start start 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.126+0000 7f4336464640 10 monclient: init 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient(hunting): picked mon.noname-b con 0x7f43300b0770 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient(hunting): picked mon.noname-a con 0x7f433014a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient(hunting): picked mon.noname-c con 0x7f43300ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300aba70 con 0x7f433014a700 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300aecb0 con 0x7f43300ac5a0 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300af150 con 0x7f43300b0770 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.127+0000 7f4336464640 10 monclient(hunting): authenticate will time out at 1278.436890s 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f43300b0770 0x7f4330149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:56562/0 (socket says 172.21.15.117:56562) 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432ffff640 1 -- 172.21.15.117:0/205112989 learned_addr learned my addr 172.21.15.117:0/205112989 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554761419 0 0) 0x7f43300af150 con 0x7f43300b0770 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.128+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f431c003520 con 0x7f43300b0770 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030175051 0 0) 0x7f43300aba70 con 0x7f433014a700 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:50.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43300af150 con 0x7f433014a700 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425291472 0 0) 0x7f43300af150 con 0x7f433014a700 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 >> v1:172.21.15.117:6790/0 conn(0x7f43300b0770 legacy=0x7f4330149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 >> v1:172.21.15.173:6789/0 conn(0x7f43300ac5a0 legacy=0x7f4330151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 monclient: found mon.noname-a 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43300ac9d0 con 0x7f433014a700 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4320001670 con 0x7f433014a700 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f432effd640 10 monclient: dump: 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.122 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f4336464640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.129+0000 7f4336464640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43200017d0 con 0x7f433014a700 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 1 -- 172.21.15.117:0/205112989 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4320001be0 con 0x7f433014a700 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f4336464640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f4336464640 4 set_mon_vals no callback set 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f432effd640 10 monclient: dump: 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.130+0000 7f4336464640 10 monclient: shutdown 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.124 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 1 -- 172.21.15.117:0/205112989 >> v1:172.21.15.117:6789/0 conn(0x7f433014a700 legacy=0x7f43300abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 1 -- 172.21.15.117:0/205112989 shutdown_connections 2024-03-29T11:26:50.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 1 -- 172.21.15.117:0/205112989 >> 172.21.15.117:0/205112989 conn(0x7f433006b660 msgr2=0x7f433006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 1 -- 172.21.15.117:0/205112989 shutdown_connections 2024-03-29T11:26:50.125 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.131+0000 7f4336464640 1 -- 172.21.15.117:0/205112989 wait complete. 2024-03-29T11:26:50.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.132+0000 7f4336464640 10 monclient: build_initial_monmap 2024-03-29T11:26:50.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.132+0000 7f4336464640 10 monclient: monmap: 2024-03-29T11:26:50.132 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:50.132 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 librados: starting msgr at 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 librados: starting objecter 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 Processor -- start 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 -- start start 2024-03-29T11:26:50.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 librados: setting wanted keys 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 1 librados: calling monclient init 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.133+0000 7f4336464640 10 monclient: init 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.134+0000 7f4336464640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.134+0000 7f4336464640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 10 monclient(hunting): picked mon.noname-a con 0x7f43300ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 10 monclient(hunting): picked mon.noname-b con 0x7f43300b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f432ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f43300ac5a0 0x7f43301deaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:53136/0 (socket says 172.21.15.117:53136) 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f432f7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f43300b0770 0x7f4330123db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:36986/0 (socket says 172.21.15.117:36986) 2024-03-29T11:26:50.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f432ffff640 1 -- 172.21.15.117:0/1205289792 learned_addr learned my addr 172.21.15.117:0/1205289792 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:50.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 10 monclient(hunting): picked mon.noname-c con 0x7f433014a700 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:50.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f432cff9640 10 client.?.objecter ms_handle_connect 0x7f43300ac5a0 2024-03-29T11:26:50.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43301459e0 con 0x7f43300ac5a0 2024-03-29T11:26:50.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4330145b70 con 0x7f43300b0770 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43301df1b0 con 0x7f433014a700 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.135+0000 7f4336464640 10 monclient(hunting): authenticate will time out at 1278.444880s 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 client.?.objecter ms_handle_connect 0x7f43300b0770 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692364804 0 0) 0x7f43301459e0 con 0x7f43300ac5a0 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4310004eb0 con 0x7f43300ac5a0 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625842327 0 0) 0x7f4330145b70 con 0x7f43300b0770 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:50.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43301459e0 con 0x7f43300b0770 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483028265 0 0) 0x7f4310004eb0 con 0x7f43300ac5a0 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4330145b70 con 0x7f43300ac5a0 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4328001cd0 con 0x7f43300ac5a0 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): dump: 2024-03-29T11:26:50.137 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 client.?.objecter ms_handle_connect 0x7f433014a700 2024-03-29T11:26:50.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2302739435 0 0) 0x7f4330145b70 con 0x7f43300ac5a0 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 >> v1:172.21.15.117:6790/0 conn(0x7f433014a700 legacy=0x7f43301244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 >> v1:172.21.15.173:6789/0 conn(0x7f43300b0770 legacy=0x7f4330123db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 monclient: found mon.a 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.136+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43301df340 con 0x7f43300ac5a0 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:50.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 10 monclient: _renew_subs 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43301df590 con 0x7f43300ac5a0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 10 monclient: _renew_subs 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43301e7030 con 0x7f43300ac5a0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f4336464640 1 librados: init done 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4328004b60 con 0x7f43300ac5a0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4328004ef0 con 0x7f43300ac5a0 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:50.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432e7fc640 4 set_mon_vals no callback set 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.137+0000 7f432cff9640 10 monclient: dump: 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.138+0000 7f4336464640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.138+0000 7f4336464640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.138+0000 7f4336464640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.138+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4314005080 con 0x7f43300ac5a0 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.140+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4328005050 con 0x7f43300ac5a0 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.140+0000 7f432f7fe640 10 monclient: get_auth_request con 0x7f431006dd10 auth_method 0 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f4328084d90 con 0x7f43300ac5a0 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 10 client.4479.objecter ms_dispatch 0x7f4330000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:26:50.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.141+0000 7f432cff9640 10 client.4479.objecter ms_handle_connect 0x7f431006dd10 2024-03-29T11:26:50.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.147+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4328050670 con 0x7f43300ac5a0 2024-03-29T11:26:50.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.147+0000 7f432cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:50.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.147+0000 7f432cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.487+0000 7f4336464640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.487+0000 7f4336464640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.487+0000 7f4336464640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.487+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4314005330 con 0x7f43300ac5a0 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.488+0000 7f432cff9640 1 -- 172.21.15.117:0/1205289792 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f43280525a0 con 0x7f43300ac5a0 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.488+0000 7f432cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.488+0000 7f432cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:50.475 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:50.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 10 librados: watch_flush enter 2024-03-29T11:26:50.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 10 librados: watch_flush exit 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 >> v1:172.21.15.117:6816/4276038175 conn(0x7f431006dd10 legacy=0x7f43100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 10 monclient: shutdown 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 >> v1:172.21.15.117:6789/0 conn(0x7f43300ac5a0 legacy=0x7f43301deaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 shutdown_connections 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 >> 172.21.15.117:0/1205289792 conn(0x7f433006b660 msgr2=0x7f43300abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 shutdown_connections 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.491+0000 7f4336464640 1 -- 172.21.15.117:0/1205289792 wait complete. 2024-03-29T11:26:50.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:50.492+0000 7f4336464640 1 librados: shutdown 2024-03-29T11:26:50.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:57.302 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:57.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.465+0000 7fc9cb015640 10 monclient: get_monmap_and_config 2024-03-29T11:26:57.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.465+0000 7fc9cb015640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.465+0000 7fc9cb015640 10 monclient: monmap: 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:26:57.466978+0000 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:26:57.466978+0000 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:26:57.455 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.468+0000 7fc9cb015640 1 Processor -- start 2024-03-29T11:26:57.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.468+0000 7fc9cb015640 1 -- start start 2024-03-29T11:26:57.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.468+0000 7fc9cb015640 10 monclient: init 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-b con 0x7fc9c40b0770 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-c con 0x7fc9c414a700 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-a con 0x7fc9c40ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40aba70 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40aecb0 con 0x7fc9c414a700 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40af150 con 0x7fc9c40b0770 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.469+0000 7fc9cb015640 10 monclient(hunting): authenticate will time out at 1285.778698s 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.470+0000 7fc9c3fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc9c414a700 0x7fc9c40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:33974/0 (socket says 172.21.15.117:33974) 2024-03-29T11:26:57.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.470+0000 7fc9c3fff640 1 -- 172.21.15.117:0/4157352740 learned_addr learned my addr 172.21.15.117:0/4157352740 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314685110 0 0) 0x7fc9c40aecb0 con 0x7fc9c414a700 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:26:57.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.471+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9ac003520 con 0x7fc9c414a700 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494386741 0 0) 0x7fc9c40aba70 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c40aecb0 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405208074 0 0) 0x7fc9c40af150 con 0x7fc9c40b0770 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.474+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c40aba70 con 0x7fc9c40b0770 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4148703131 0 0) 0x7fc9ac003520 con 0x7fc9c414a700 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 >> v1:172.21.15.117:6790/0 conn(0x7fc9c40b0770 legacy=0x7fc9c4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 >> v1:172.21.15.117:6789/0 conn(0x7fc9c40ac5a0 legacy=0x7fc9c4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 monclient: found mon.noname-c 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40ac9d0 con 0x7fc9c414a700 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9b80016b0 con 0x7fc9c414a700 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: dump: 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9b8001810 con 0x7fc9c414a700 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 1 -- 172.21.15.117:0/4157352740 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9b8001be0 con 0x7fc9c414a700 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9c37fe640 10 monclient: dump: 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:57.466 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9cb015640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.475+0000 7fc9cb015640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:57.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 4 set_mon_vals no callback set 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 10 monclient: shutdown 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 1 -- 172.21.15.117:0/4157352740 >> v1:172.21.15.173:6789/0 conn(0x7fc9c414a700 legacy=0x7fc9c40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 1 -- 172.21.15.117:0/4157352740 shutdown_connections 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 1 -- 172.21.15.117:0/4157352740 >> 172.21.15.117:0/4157352740 conn(0x7fc9c406b660 msgr2=0x7fc9c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 1 -- 172.21.15.117:0/4157352740 shutdown_connections 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.476+0000 7fc9cb015640 1 -- 172.21.15.117:0/4157352740 wait complete. 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.477+0000 7fc9cb015640 10 monclient: build_initial_monmap 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.477+0000 7fc9cb015640 10 monclient: monmap: 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:57.470 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 librados: starting msgr at 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 librados: starting objecter 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 Processor -- start 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 -- start start 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 librados: setting wanted keys 2024-03-29T11:26:57.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 1 librados: calling monclient init 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.478+0000 7fc9cb015640 10 monclient: init 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.479+0000 7fc9cb015640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.479+0000 7fc9cb015640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.479+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-b con 0x7fc9c40ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.479+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-c con 0x7fc9c40b0770 addr v1:172.21.15.117:6790/0 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 10 monclient(hunting): picked mon.noname-a con 0x7fc9c414a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c41dfde0 con 0x7fc9c414a700 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c41dfa20 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c41dfbb0 con 0x7fc9c40b0770 2024-03-29T11:26:57.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:57.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.481+0000 7fc9cb015640 10 monclient(hunting): authenticate will time out at 1285.790205s 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c8db1640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc9c40ac5a0 0x7fc9c41df310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:36608/0 (socket says 172.21.15.117:36608) 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c8db1640 1 -- 172.21.15.117:0/1196163282 learned_addr learned my addr 172.21.15.117:0/1196163282 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 client.?.objecter ms_handle_connect 0x7fc9c40ac5a0 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931416041 0 0) 0x7fc9c41dfa20 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.482+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b0004eb0 con 0x7fc9c40ac5a0 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 client.?.objecter ms_handle_connect 0x7fc9c40b0770 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052482570 0 0) 0x7fc9c41dfbb0 con 0x7fc9c40b0770 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:26:57.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c41dfa20 con 0x7fc9c40b0770 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3767773294 0 0) 0x7fc9c41dfa20 con 0x7fc9c40b0770 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9b0005490 con 0x7fc9c40b0770 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9b8001b20 con 0x7fc9c40b0770 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): dump: 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.477 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.483+0000 7fc9c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3862468622 0 0) 0x7fc9b0005490 con 0x7fc9c40b0770 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 >> v1:172.21.15.173:6789/0 conn(0x7fc9c40ac5a0 legacy=0x7fc9c41df310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 >> v1:172.21.15.117:6789/0 conn(0x7fc9c414a700 legacy=0x7fc9c4126b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 monclient: found mon.c 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:57.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c4145130 con 0x7fc9c40b0770 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 10 monclient: _renew_subs 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4145380 con 0x7fc9c40b0770 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 10 monclient: _renew_subs 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c41e77d0 con 0x7fc9c40b0770 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9cb015640 1 librados: init done 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9b8001d00 con 0x7fc9c40b0770 2024-03-29T11:26:57.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.485+0000 7fc9c2ffd640 4 set_mon_vals no callback set 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.486+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fc9b8001b20 con 0x7fc9c40b0770 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.486+0000 7fc9c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.486+0000 7fc9c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.486+0000 7fc9c17fa640 10 monclient: dump: 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:57.480 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.487+0000 7fc9cb015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.487+0000 7fc9cb015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.487+0000 7fc9cb015640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:57.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.487+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a4005080 con 0x7fc9c40b0770 2024-03-29T11:26:57.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.489+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fc9b801a900 con 0x7fc9c40b0770 2024-03-29T11:26:57.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.489+0000 7fc9c8db1640 10 monclient: get_auth_request con 0x7fc9b006e2a0 auth_method 0 2024-03-29T11:26:57.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 10 client.4586.objecter ms_handle_connect 0x7fc9b006e2a0 2024-03-29T11:26:57.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fc9b8054230 con 0x7fc9c40b0770 2024-03-29T11:26:57.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 10 client.4586.objecter ms_dispatch 0x7fc9c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:57.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:57.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:57.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.490+0000 7fc9c17fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:26:57.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.496+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9b80504d0 con 0x7fc9c40b0770 2024-03-29T11:26:57.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.496+0000 7fc9c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:57.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.496+0000 7fc9c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:57.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.823+0000 7fc9cb015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.823+0000 7fc9cb015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.823+0000 7fc9cb015640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:26:57.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.823+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a4005330 con 0x7fc9c40b0770 2024-03-29T11:26:57.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.827+0000 7fc9c17fa640 1 -- 172.21.15.117:0/1196163282 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fc9b8054e60 con 0x7fc9c40b0770 2024-03-29T11:26:57.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.827+0000 7fc9c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:57.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.827+0000 7fc9c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:57.815 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:26:57.815 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 10 librados: watch_flush enter 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 10 librados: watch_flush exit 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 >> v1:172.21.15.117:6816/4276038175 conn(0x7fc9b006e2a0 legacy=0x7fc9b0070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 10 monclient: shutdown 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:57.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.832+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 >> v1:172.21.15.117:6790/0 conn(0x7fc9c40b0770 legacy=0x7fc9c4126480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:57.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.834+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 shutdown_connections 2024-03-29T11:26:57.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.834+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 >> 172.21.15.117:0/1196163282 conn(0x7fc9c406b660 msgr2=0x7fc9c40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:57.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.834+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 shutdown_connections 2024-03-29T11:26:57.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.834+0000 7fc9cb015640 1 -- 172.21.15.117:0/1196163282 wait complete. 2024-03-29T11:26:57.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:26:57.835+0000 7fc9cb015640 1 librados: shutdown 2024-03-29T11:26:57.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:04.646 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.785+0000 7f7bafe3e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.785+0000 7f7bafe3e640 10 monclient: build_initial_monmap 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.785+0000 7f7bafe3e640 10 monclient: monmap: 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:04.786657+0000 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:04.786657+0000 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.773 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:04.774 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:04.774 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:04.774 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.786+0000 7f7bafe3e640 1 Processor -- start 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 1 -- start start 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient: init 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba81457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba813f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba80a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80aaef0 con 0x7f7ba81457f0 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8145bd0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8145eb0 con 0x7f7ba813f650 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:04.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.787+0000 7f7bafe3e640 10 monclient(hunting): authenticate will time out at 1293.096861s 2024-03-29T11:27:04.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.788+0000 7f7bae3db640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7ba80a9d70 0x7f7ba814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:58198/0 (socket says 172.21.15.117:58198) 2024-03-29T11:27:04.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.788+0000 7f7bae3db640 1 -- 172.21.15.117:0/1535931214 learned_addr learned my addr 172.21.15.117:0/1535931214 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:04.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502109867 0 0) 0x7f7ba8145eb0 con 0x7f7ba813f650 2024-03-29T11:27:04.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90003520 con 0x7f7ba813f650 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308828250 0 0) 0x7f7ba8145bd0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.789+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba8145eb0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571600967 0 0) 0x7f7ba8145eb0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 >> v1:172.21.15.117:6790/0 conn(0x7f7ba813f650 legacy=0x7f7ba80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 >> v1:172.21.15.117:6789/0 conn(0x7f7ba81457f0 legacy=0x7f7ba813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: _finish_hunting 0 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 monclient: found mon.noname-c 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8146340 con 0x7f7ba80a9d70 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: _finish_auth 0 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: _check_auth_tickets 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7ba00015f0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bafe3e640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: dump: 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:04.781 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bafe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba0001750 con 0x7f7ba80a9d70 2024-03-29T11:27:04.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 1 -- 172.21.15.117:0/1535931214 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7ba0001ba0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bacbd8640 10 monclient: dump: 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:04.783 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bafe3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.790+0000 7f7bafe3e640 4 set_mon_vals no callback set 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 10 monclient: shutdown 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 1 -- 172.21.15.117:0/1535931214 >> v1:172.21.15.173:6789/0 conn(0x7f7ba80a9d70 legacy=0x7f7ba814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 1 -- 172.21.15.117:0/1535931214 shutdown_connections 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 1 -- 172.21.15.117:0/1535931214 >> 172.21.15.117:0/1535931214 conn(0x7f7ba8066a00 msgr2=0x7f7ba8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 1 -- 172.21.15.117:0/1535931214 shutdown_connections 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.791+0000 7f7bafe3e640 1 -- 172.21.15.117:0/1535931214 wait complete. 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.792+0000 7f7bafe3e640 10 monclient: build_initial_monmap 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.792+0000 7f7bafe3e640 10 monclient: monmap: 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:04.784 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:04.785 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 librados: starting msgr at 2024-03-29T11:27:04.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 librados: starting objecter 2024-03-29T11:27:04.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 Processor -- start 2024-03-29T11:27:04.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 -- start start 2024-03-29T11:27:04.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 librados: setting wanted keys 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 1 librados: calling monclient init 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.793+0000 7f7bafe3e640 10 monclient: init 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.794+0000 7f7bafe3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.794+0000 7f7bafe3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.794+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba80a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba813f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bad3d9640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7ba813f650 0x7f7ba81db620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:59314/0 (socket says 172.21.15.117:59314) 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bad3d9640 1 -- 172.21.15.117:0/2818932828 learned_addr learned my addr 172.21.15.117:0/2818932828 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba813b1a0 con 0x7f7ba80a9d70 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba813ab50 con 0x7f7ba81457f0 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7ba67fc640 10 client.?.objecter ms_handle_connect 0x7f7ba813f650 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8139930 con 0x7f7ba813f650 2024-03-29T11:27:04.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.795+0000 7f7bafe3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7bafe3e640 10 monclient(hunting): authenticate will time out at 1293.105101s 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 client.?.objecter ms_handle_connect 0x7f7ba81457f0 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506581766 0 0) 0x7f7ba813ab50 con 0x7f7ba81457f0 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.796+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b94004eb0 con 0x7f7ba81457f0 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384834750 0 0) 0x7f7ba8139930 con 0x7f7ba813f650 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:27:04.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba813ab50 con 0x7f7ba813f650 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912824193 0 0) 0x7f7b94004eb0 con 0x7f7ba81457f0 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba8139930 con 0x7f7ba81457f0 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7ba0001a60 con 0x7f7ba81457f0 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): dump: 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:04.794 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1146353582 0 0) 0x7f7ba813ab50 con 0x7f7ba813f650 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b94004eb0 con 0x7f7ba813f650 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 693925477 0 0) 0x7f7ba8139930 con 0x7f7ba81457f0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 >> v1:172.21.15.117:6790/0 conn(0x7f7ba813f650 legacy=0x7f7ba81db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.797+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 >> v1:172.21.15.117:6789/0 conn(0x7f7ba80a9d70 legacy=0x7f7ba8138a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 1 monclient: found mon.b 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8139ac0 con 0x7f7ba81457f0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:04.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7ba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 10 monclient: _renew_subs 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba8142a70 con 0x7f7ba81457f0 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 10 monclient: _renew_subs 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba813b630 con 0x7f7ba81457f0 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.799+0000 7f7bafe3e640 1 librados: init done 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.800+0000 7f7bafe3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.796 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.800+0000 7f7bafe3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.800+0000 7f7bafe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.800+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8075f10 con 0x7f7ba81457f0 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba0001c20 con 0x7f7ba81457f0 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7ba0004c30 con 0x7f7ba81457f0 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba67fc640 10 monclient: dump: 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:04.797 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.801+0000 7f7ba7fff640 4 set_mon_vals no callback set 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.802+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7ba001ac00 con 0x7f7ba81457f0 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f7ba00848e0 con 0x7f7ba81457f0 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 10 client.4495.objecter ms_dispatch 0x7f7ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7badbda640 10 monclient: get_auth_request con 0x7f7b9406dd90 auth_method 0 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:27:04.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.803+0000 7f7ba67fc640 10 client.4495.objecter ms_handle_connect 0x7f7b9406dd90 2024-03-29T11:27:04.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.810+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ba00501c0 con 0x7f7ba81457f0 2024-03-29T11:27:04.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.810+0000 7f7ba67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:04.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:04.810+0000 7f7ba67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:05.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.122+0000 7f7bafe3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.122+0000 7f7bafe3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.122+0000 7f7bafe3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:05.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.122+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ba813b350 con 0x7f7ba81457f0 2024-03-29T11:27:05.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.123+0000 7f7ba67fc640 1 -- 172.21.15.117:0/2818932828 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f7ba813b350 con 0x7f7ba81457f0 2024-03-29T11:27:05.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.123+0000 7f7ba67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:05.110 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:05.111 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:05.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.123+0000 7f7ba67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 10 librados: watch_flush enter 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 10 librados: watch_flush exit 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 >> v1:172.21.15.117:6816/4276038175 conn(0x7f7b9406dd90 legacy=0x7f7b94070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 10 monclient: shutdown 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 >> v1:172.21.15.173:6789/0 conn(0x7f7ba81457f0 legacy=0x7f7ba81410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 shutdown_connections 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 >> 172.21.15.117:0/2818932828 conn(0x7f7ba8066a00 msgr2=0x7f7ba80b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:05.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 shutdown_connections 2024-03-29T11:27:05.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 -- 172.21.15.117:0/2818932828 wait complete. 2024-03-29T11:27:05.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:05.126+0000 7f7bafe3e640 1 librados: shutdown 2024-03-29T11:27:05.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:11.937 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.093+0000 7fa84b7b5640 10 monclient: get_monmap_and_config 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.093+0000 7fa84b7b5640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.093+0000 7fa84b7b5640 10 monclient: monmap: 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:12.094549+0000 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:12.094549+0000 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.081 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.082 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:12.082 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:12.082 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:12.082 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.084 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.097+0000 7fa84b7b5640 1 Processor -- start 2024-03-29T11:27:12.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.097+0000 7fa84b7b5640 1 -- start start 2024-03-29T11:27:12.085 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.097+0000 7fa84b7b5640 10 monclient: init 2024-03-29T11:27:12.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa844070120 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa8440b0420 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa844070500 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa844072090 con 0x7fa8440b0420 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa844072370 con 0x7fa844070120 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa844072650 con 0x7fa844070500 2024-03-29T11:27:12.088 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa84b7b5640 10 monclient(hunting): authenticate will time out at 1300.407415s 2024-03-29T11:27:12.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa849d52640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fa844070500 0x7fa84414b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46782/0 (socket says 172.21.15.117:46782) 2024-03-29T11:27:12.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.098+0000 7fa849d52640 1 -- 172.21.15.117:0/2560813138 learned_addr learned my addr 172.21.15.117:0/2560813138 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:12.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220083604 0 0) 0x7fa844072650 con 0x7fa844070500 2024-03-29T11:27:12.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa830003520 con 0x7fa844070500 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006884064 0 0) 0x7fa844072090 con 0x7fa8440b0420 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.101+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa844072650 con 0x7fa8440b0420 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428932888 0 0) 0x7fa844072650 con 0x7fa8440b0420 2024-03-29T11:27:12.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 >> v1:172.21.15.117:6790/0 conn(0x7fa844070500 legacy=0x7fa84414b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 >> v1:172.21.15.173:6789/0 conn(0x7fa844070120 legacy=0x7fa84413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:12.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 monclient: found mon.noname-a 2024-03-29T11:27:12.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844140090 con 0x7fa8440b0420 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa8340015a0 con 0x7fa8440b0420 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: dump: 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.099 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.100 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.100 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:12.100 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:12.100 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa84b7b5640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa84b7b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa834001700 con 0x7fa8440b0420 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 1 -- 172.21.15.117:0/2560813138 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa834001ad0 con 0x7fa8440b0420 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa83bfff640 10 monclient: dump: 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.102 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa84b7b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.102+0000 7fa84b7b5640 4 set_mon_vals no callback set 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.104+0000 7fa84b7b5640 10 monclient: shutdown 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.104+0000 7fa84b7b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.104+0000 7fa84b7b5640 1 -- 172.21.15.117:0/2560813138 >> v1:172.21.15.117:6789/0 conn(0x7fa8440b0420 legacy=0x7fa8441405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.108+0000 7fa84b7b5640 1 -- 172.21.15.117:0/2560813138 shutdown_connections 2024-03-29T11:27:12.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.108+0000 7fa84b7b5640 1 -- 172.21.15.117:0/2560813138 >> 172.21.15.117:0/2560813138 conn(0x7fa844067500 msgr2=0x7fa8440b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.108+0000 7fa84b7b5640 1 -- 172.21.15.117:0/2560813138 shutdown_connections 2024-03-29T11:27:12.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.109+0000 7fa84b7b5640 1 -- 172.21.15.117:0/2560813138 wait complete. 2024-03-29T11:27:12.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.110+0000 7fa84b7b5640 10 monclient: build_initial_monmap 2024-03-29T11:27:12.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.110+0000 7fa84b7b5640 10 monclient: monmap: 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.110+0000 7fa84b7b5640 1 librados: starting msgr at 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.110+0000 7fa84b7b5640 1 librados: starting objecter 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.111+0000 7fa84b7b5640 1 Processor -- start 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.111+0000 7fa84b7b5640 1 -- start start 2024-03-29T11:27:12.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.111+0000 7fa84b7b5640 1 librados: setting wanted keys 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.111+0000 7fa84b7b5640 1 librados: calling monclient init 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.111+0000 7fa84b7b5640 10 monclient: init 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.112+0000 7fa84b7b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.112+0000 7fa84b7b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-b con 0x7fa844070120 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-a con 0x7fa844070500 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 10 monclient(hunting): picked mon.noname-c con 0x7fa8440b0420 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8440af500 con 0x7fa844070500 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa844133e30 con 0x7fa844070120 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8441e02c0 con 0x7fa8440b0420 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.113+0000 7fa84b7b5640 10 monclient(hunting): authenticate will time out at 1300.422207s 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa849551640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fa844070120 0x7fa8441df4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:41497/0 (socket says 172.21.15.117:41497) 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa849551640 1 -- 172.21.15.117:0/3368445471 learned_addr learned my addr 172.21.15.117:0/3368445471 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:12.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa839ffb640 10 client.?.objecter ms_handle_connect 0x7fa844070120 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa839ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa839ffb640 10 client.?.objecter ms_handle_connect 0x7fa8440b0420 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.117+0000 7fa839ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004023179 0 0) 0x7fa8441e02c0 con 0x7fa8440b0420 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa828004eb0 con 0x7fa8440b0420 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081300520 0 0) 0x7fa844133e30 con 0x7fa844070120 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:27:12.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8441e02c0 con 0x7fa844070120 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.118+0000 7fa839ffb640 10 client.?.objecter ms_handle_connect 0x7fa844070500 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 967156940 0 0) 0x7fa8441e02c0 con 0x7fa844070120 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa844133e30 con 0x7fa844070120 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa83c0043c0 con 0x7fa844070120 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.114 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186562818 0 0) 0x7fa8440af500 con 0x7fa844070500 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8441e02c0 con 0x7fa844070500 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2701571134 0 0) 0x7fa844133e30 con 0x7fa844070120 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 >> v1:172.21.15.117:6790/0 conn(0x7fa8440b0420 legacy=0x7fa8441dfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 >> v1:172.21.15.117:6789/0 conn(0x7fa844070500 legacy=0x7fa8440aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:12.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 monclient: found mon.b 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844133a10 con 0x7fa844070120 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.119+0000 7fa839ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 10 monclient: _renew_subs 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8440af770 con 0x7fa844070120 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa83c004520 con 0x7fa844070120 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fa83c0014b0 con 0x7fa844070120 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:12.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa839ffb640 10 monclient: dump: 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa83b7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 10 monclient: _renew_subs 2024-03-29T11:27:12.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8440ad380 con 0x7fa844070120 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.120+0000 7fa84b7b5640 1 librados: init done 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.122+0000 7fa84b7b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.123+0000 7fa84b7b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.123+0000 7fa84b7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:12.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.123+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844075f10 con 0x7fa844070120 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.124+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fa83c001610 con 0x7fa844070120 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fa83c084d80 con 0x7fa844070120 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa839ffb640 10 client.4498.objecter ms_dispatch 0x7fa844000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa839ffb640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa839ffb640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa839ffb640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.125+0000 7fa848d50640 10 monclient: get_auth_request con 0x7fa82806ddf0 auth_method 0 2024-03-29T11:27:12.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.126+0000 7fa839ffb640 10 client.4498.objecter ms_handle_connect 0x7fa82806ddf0 2024-03-29T11:27:12.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.132+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa83c050490 con 0x7fa844070120 2024-03-29T11:27:12.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.132+0000 7fa839ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:12.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.132+0000 7fa839ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:12.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.441+0000 7fa84b7b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.441+0000 7fa84b7b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.441+0000 7fa84b7b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:12.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.441+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa844070f50 con 0x7fa844070120 2024-03-29T11:27:12.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.442+0000 7fa839ffb640 1 -- 172.21.15.117:0/3368445471 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fa83c052590 con 0x7fa844070120 2024-03-29T11:27:12.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.442+0000 7fa839ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:12.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.442+0000 7fa839ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:12.431 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:12.431 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:12.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.446+0000 7fa84b7b5640 10 librados: watch_flush enter 2024-03-29T11:27:12.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 10 librados: watch_flush exit 2024-03-29T11:27:12.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 >> v1:172.21.15.117:6816/4276038175 conn(0x7fa82806ddf0 legacy=0x7fa8280702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 10 monclient: shutdown 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.447+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 >> v1:172.21.15.173:6789/0 conn(0x7fa844070120 legacy=0x7fa8441df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.448+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 shutdown_connections 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.448+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 >> 172.21.15.117:0/3368445471 conn(0x7fa844067500 msgr2=0x7fa84406d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.448+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 shutdown_connections 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.448+0000 7fa84b7b5640 1 -- 172.21.15.117:0/3368445471 wait complete. 2024-03-29T11:27:12.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:12.448+0000 7fa84b7b5640 1 librados: shutdown 2024-03-29T11:27:12.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:19.256 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.416+0000 7ffa7cb6c640 10 monclient: get_monmap_and_config 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.416+0000 7ffa7cb6c640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.416+0000 7ffa7cb6c640 10 monclient: monmap: 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:19.417498+0000 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:19.417498+0000 2024-03-29T11:27:19.404 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.405 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.405 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:19.405 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:19.405 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:19.405 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.418+0000 7ffa7cb6c640 1 Processor -- start 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.419+0000 7ffa7cb6c640 1 -- start start 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.419+0000 7ffa7cb6c640 10 monclient: init 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa780b1f60 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa78071eb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa78143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:19.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7806ef80 con 0x7ffa78071eb0 2024-03-29T11:27:19.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7806f260 con 0x7ffa78143d10 2024-03-29T11:27:19.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa780b40c0 con 0x7ffa780b1f60 2024-03-29T11:27:19.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.420+0000 7ffa7cb6c640 10 monclient(hunting): authenticate will time out at 1307.729330s 2024-03-29T11:27:19.412 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.425+0000 7ffa7659c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffa780b1f60 0x7ffa78143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34638/0 (socket says 172.21.15.117:34638) 2024-03-29T11:27:19.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.425+0000 7ffa7659c640 1 -- 172.21.15.117:0/461102289 learned_addr learned my addr 172.21.15.117:0/461102289 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359158385 0 0) 0x7ffa780b40c0 con 0x7ffa780b1f60 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa60003520 con 0x7ffa780b1f60 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360747601 0 0) 0x7ffa7806ef80 con 0x7ffa78071eb0 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:27:19.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.426+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa780b40c0 con 0x7ffa78071eb0 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179978240 0 0) 0x7ffa780b40c0 con 0x7ffa78071eb0 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 >> v1:172.21.15.117:6790/0 conn(0x7ffa780b1f60 legacy=0x7ffa78143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 >> v1:172.21.15.173:6789/0 conn(0x7ffa78143d10 legacy=0x7ffa78146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 monclient: found mon.noname-a 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:27:19.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa78155d70 con 0x7ffa78071eb0 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa700019f0 con 0x7ffa78071eb0 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7559a640 10 monclient: dump: 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:19.416 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7cb6c640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.427+0000 7ffa7cb6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa70001c10 con 0x7ffa78071eb0 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 1 -- 172.21.15.117:0/461102289 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa70001fa0 con 0x7ffa78071eb0 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7cb6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7cb6c640 4 set_mon_vals no callback set 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.429+0000 7ffa7559a640 10 monclient: dump: 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:19.418 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.431+0000 7ffa7cb6c640 10 monclient: shutdown 2024-03-29T11:27:19.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.431+0000 7ffa7cb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:19.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.431+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/461102289 >> v1:172.21.15.117:6789/0 conn(0x7ffa78071eb0 legacy=0x7ffa78146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.431+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/461102289 shutdown_connections 2024-03-29T11:27:19.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.431+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/461102289 >> 172.21.15.117:0/461102289 conn(0x7ffa78066a00 msgr2=0x7ffa78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:19.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.433+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/461102289 shutdown_connections 2024-03-29T11:27:19.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.435+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/461102289 wait complete. 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.436+0000 7ffa7cb6c640 10 monclient: build_initial_monmap 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.436+0000 7ffa7cb6c640 10 monclient: monmap: 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:19.425 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.436+0000 7ffa7cb6c640 1 librados: starting msgr at 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.436+0000 7ffa7cb6c640 1 librados: starting objecter 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.436+0000 7ffa7cb6c640 1 Processor -- start 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.437+0000 7ffa7cb6c640 1 -- start start 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.437+0000 7ffa7cb6c640 1 librados: setting wanted keys 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.437+0000 7ffa7cb6c640 1 librados: calling monclient init 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.437+0000 7ffa7cb6c640 10 monclient: init 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:19.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa780b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:19.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa78143d10 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa780ab870 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa780abd90 con 0x7ffa780b1f60 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa780ac2b0 con 0x7ffa780ab870 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa78140aa0 con 0x7ffa78143d10 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:19.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.438+0000 7ffa7cb6c640 10 monclient(hunting): authenticate will time out at 1307.747187s 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.441+0000 7ffa75d9b640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffa78143d10 0x7ffa7813fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53984/0 (socket says 172.21.15.117:53984) 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.441+0000 7ffa75d9b640 1 -- 172.21.15.117:0/763376902 learned_addr learned my addr 172.21.15.117:0/763376902 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 client.?.objecter ms_handle_connect 0x7ffa78143d10 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604011329 0 0) 0x7ffa78140aa0 con 0x7ffa78143d10 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58004eb0 con 0x7ffa78143d10 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 client.?.objecter ms_handle_connect 0x7ffa780b1f60 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690445269 0 0) 0x7ffa780abd90 con 0x7ffa780b1f60 2024-03-29T11:27:19.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.442+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78140aa0 con 0x7ffa780b1f60 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450146433 0 0) 0x7ffa58004eb0 con 0x7ffa78143d10 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa780abd90 con 0x7ffa78143d10 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa700019f0 con 0x7ffa78143d10 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.437 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 client.?.objecter ms_handle_connect 0x7ffa780ab870 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064526693 0 0) 0x7ffa780ac2b0 con 0x7ffa780ab870 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58004eb0 con 0x7ffa780ab870 2024-03-29T11:27:19.438 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.443+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3281002261 0 0) 0x7ffa780abd90 con 0x7ffa78143d10 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 >> v1:172.21.15.173:6789/0 conn(0x7ffa780ab870 legacy=0x7ffa781ef7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 >> v1:172.21.15.117:6789/0 conn(0x7ffa780b1f60 legacy=0x7ffa781e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 monclient: found mon.c 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa780aad40 con 0x7ffa78143d10 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 10 monclient: _renew_subs 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:19.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa780aaff0 con 0x7ffa78143d10 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 10 monclient: _renew_subs 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa781e4880 con 0x7ffa78143d10 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa7cb6c640 1 librados: init done 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa70003c20 con 0x7ffa78143d10 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa70004030 con 0x7ffa78143d10 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.444+0000 7ffa66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.445+0000 7ffa66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.445+0000 7ffa66ffd640 10 monclient: dump: 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.440 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:19.441 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.445+0000 7ffa74d99640 4 set_mon_vals no callback set 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7ffa70004190 con 0x7ffa78143d10 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7ffa70084e50 con 0x7ffa78143d10 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 10 client.4604.objecter ms_dispatch 0x7ffa78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa66ffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.447+0000 7ffa7659c640 10 monclient: get_auth_request con 0x7ffa5806dea0 auth_method 0 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.448+0000 7ffa66ffd640 10 client.4604.objecter ms_handle_connect 0x7ffa5806dea0 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.450+0000 7ffa7cb6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.442 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.450+0000 7ffa7cb6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.450+0000 7ffa7cb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:19.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.450+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c005080 con 0x7ffa78143d10 2024-03-29T11:27:19.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.458+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa70050730 con 0x7ffa78143d10 2024-03-29T11:27:19.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.458+0000 7ffa66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:19.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.458+0000 7ffa66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:19.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.786+0000 7ffa7cb6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:19.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.786+0000 7ffa7cb6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:19.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.786+0000 7ffa7cb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:19.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.786+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa5c005330 con 0x7ffa78143d10 2024-03-29T11:27:19.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.788+0000 7ffa66ffd640 1 -- 172.21.15.117:0/763376902 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7ffa70052660 con 0x7ffa78143d10 2024-03-29T11:27:19.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.788+0000 7ffa66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:19.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.788+0000 7ffa66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:19.776 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:19.776 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:19.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 10 librados: watch_flush enter 2024-03-29T11:27:19.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 10 librados: watch_flush exit 2024-03-29T11:27:19.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:27:19.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 >> v1:172.21.15.117:6816/4276038175 conn(0x7ffa5806dea0 legacy=0x7ffa58070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 10 monclient: shutdown 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 >> v1:172.21.15.117:6790/0 conn(0x7ffa78143d10 legacy=0x7ffa7813fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 shutdown_connections 2024-03-29T11:27:19.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.791+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 >> 172.21.15.117:0/763376902 conn(0x7ffa78066a00 msgr2=0x7ffa780b6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:19.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.794+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 shutdown_connections 2024-03-29T11:27:19.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.794+0000 7ffa7cb6c640 1 -- 172.21.15.117:0/763376902 wait complete. 2024-03-29T11:27:19.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:19.795+0000 7ffa7cb6c640 1 librados: shutdown 2024-03-29T11:27:19.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:26.609 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.766+0000 7f931bf96640 10 monclient: get_monmap_and_config 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.766+0000 7f931bf96640 10 monclient: build_initial_monmap 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.766+0000 7f931bf96640 10 monclient: monmap: 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:26.767696+0000 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:26.767696+0000 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:26.755 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.769+0000 7f931bf96640 1 Processor -- start 2024-03-29T11:27:26.757 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.770+0000 7f931bf96640 1 -- start start 2024-03-29T11:27:26.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.770+0000 7f931bf96640 10 monclient: init 2024-03-29T11:27:26.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:26.758 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-b con 0x7f93141457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-a con 0x7f931413f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-c con 0x7f93140a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93140aaef0 con 0x7f931413f650 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314145bd0 con 0x7f93140a9d70 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314145eb0 con 0x7f93141457f0 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:26.759 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.771+0000 7f931bf96640 10 monclient(hunting): authenticate will time out at 1315.080195s 2024-03-29T11:27:26.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.774+0000 7f9319531640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f931413f650 0x7f93140b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59673/0 (socket says 172.21.15.117:59673) 2024-03-29T11:27:26.762 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.774+0000 7f9319531640 1 -- 172.21.15.117:0/2514782522 learned_addr learned my addr 172.21.15.117:0/2514782522 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248094043 0 0) 0x7f93140aaef0 con 0x7f931413f650 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _negotiate 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9300003520 con 0x7f931413f650 2024-03-29T11:27:26.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085534709 0 0) 0x7f9314145eb0 con 0x7f93141457f0 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _negotiate 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93140aaef0 con 0x7f93141457f0 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4243597068 0 0) 0x7f9300003520 con 0x7f931413f650 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 >> v1:172.21.15.117:6790/0 conn(0x7f93141457f0 legacy=0x7f931413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 >> v1:172.21.15.173:6789/0 conn(0x7f93140a9d70 legacy=0x7f931414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: _finish_hunting 0 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 monclient: found mon.noname-a 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:26.765 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314146340 con 0x7f931413f650 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: _finish_auth 0 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: _check_auth_tickets 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9304001690 con 0x7f931413f650 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.775+0000 7f9318d30640 10 monclient: dump: 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:26.766 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f931bf96640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f931bf96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f9318d30640 1 -- 172.21.15.117:0/2514782522 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93040017f0 con 0x7f931413f650 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f9318d30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:26.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f931bf96640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.776+0000 7f931bf96640 4 set_mon_vals no callback set 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 10 monclient: shutdown 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 1 -- 172.21.15.117:0/2514782522 >> v1:172.21.15.117:6789/0 conn(0x7f931413f650 legacy=0x7f93140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 1 -- 172.21.15.117:0/2514782522 shutdown_connections 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 1 -- 172.21.15.117:0/2514782522 >> 172.21.15.117:0/2514782522 conn(0x7f9314066a00 msgr2=0x7f9314066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 1 -- 172.21.15.117:0/2514782522 shutdown_connections 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.777+0000 7f931bf96640 1 -- 172.21.15.117:0/2514782522 wait complete. 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.778+0000 7f931bf96640 10 monclient: build_initial_monmap 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.778+0000 7f931bf96640 10 monclient: monmap: 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:26.768 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 librados: starting msgr at 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 librados: starting objecter 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 Processor -- start 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 -- start start 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 librados: setting wanted keys 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 1 librados: calling monclient init 2024-03-29T11:27:26.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.779+0000 7f931bf96640 10 monclient: init 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-a con 0x7f93140a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-c con 0x7f93141457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient(hunting): picked mon.noname-b con 0x7f93140700c0 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931406fba0 con 0x7f93140a9d70 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93141d3070 con 0x7f93140700c0 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931406efc0 con 0x7f93141457f0 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.780+0000 7f931bf96640 10 monclient(hunting): authenticate will time out at 1315.089142s 2024-03-29T11:27:26.770 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.781+0000 7f9319531640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f93141457f0 0x7f9314139cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38688/0 (socket says 172.21.15.117:38688) 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.781+0000 7f9319531640 1 -- 172.21.15.117:0/2314789935 learned_addr learned my addr 172.21.15.117:0/2314789935 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.781+0000 7f930a7fc640 10 client.?.objecter ms_handle_connect 0x7f93141457f0 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.781+0000 7f930a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.781+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325856151 0 0) 0x7f931406efc0 con 0x7f93141457f0 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.782+0000 7f930a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.782+0000 7f930a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.782+0000 7f930a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.782+0000 7f930a7fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:27:26.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.782+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f8004eb0 con 0x7f93141457f0 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 client.?.objecter ms_handle_connect 0x7f93140a9d70 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732833556 0 0) 0x7f92f8004eb0 con 0x7f93141457f0 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f931406efc0 con 0x7f93141457f0 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768104011 0 0) 0x7f931406fba0 con 0x7f93140a9d70 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:26.772 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f8004eb0 con 0x7f93140a9d70 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f9304004980 con 0x7f93141457f0 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:26.773 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1627660334 0 0) 0x7f92f8004eb0 con 0x7f93140a9d70 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f931406fba0 con 0x7f93140a9d70 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f930c001a30 con 0x7f93140a9d70 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:26.774 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.775 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:26.775 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:26.775 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:26.775 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296950788 0 0) 0x7f931406fba0 con 0x7f93140a9d70 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 >> v1:172.21.15.117:6790/0 conn(0x7f93141457f0 legacy=0x7f9314139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.784+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 >> v1:172.21.15.173:6789/0 conn(0x7f93140700c0 legacy=0x7f93141de1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 1 monclient: found mon.a 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931406f7c0 con 0x7f93140a9d70 2024-03-29T11:27:26.778 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:26.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:26.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f930a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:26.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f931bf96640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:27:26.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f931bf96640 10 monclient: _renew_subs 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f931bf96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.785+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9314141880 con 0x7f93140a9d70 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f931bf96640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f931bf96640 10 monclient: _renew_subs 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f931bf96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f931413a5e0 con 0x7f93140a9d70 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f931bf96640 1 librados: init done 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f930c001f20 con 0x7f93140a9d70 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f930c003b10 con 0x7f93140a9d70 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:26.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.786+0000 7f930a7fc640 10 monclient: dump: 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.787+0000 7f931bf96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:26.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.787+0000 7f931bf96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:26.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.787+0000 7f931bf96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:26.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.787+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc005080 con 0x7f93140a9d70 2024-03-29T11:27:26.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.796+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f930c019ae0 con 0x7f93140a9d70 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f930c083900 con 0x7f93140a9d70 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 10 client.4503.objecter ms_dispatch 0x7f9314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f930c0b08b0 con 0x7f93140a9d70 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f9319531640 10 monclient: get_auth_request con 0x7f92f806dd90 auth_method 0 2024-03-29T11:27:26.789 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:26.800+0000 7f930a7fc640 10 client.4503.objecter ms_handle_connect 0x7f92f806dd90 2024-03-29T11:27:27.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.119+0000 7f931bf96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.119+0000 7f931bf96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.119+0000 7f931bf96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:27.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.119+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92fc005330 con 0x7f93140a9d70 2024-03-29T11:27:27.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.120+0000 7f930a7fc640 1 -- 172.21.15.117:0/2314789935 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f930c04f1e0 con 0x7f93140a9d70 2024-03-29T11:27:27.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.120+0000 7f930a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:27.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.120+0000 7f930a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:27.110 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 10 librados: watch_flush enter 2024-03-29T11:27:27.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 10 librados: watch_flush exit 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 >> v1:172.21.15.117:6816/4276038175 conn(0x7f92f806dd90 legacy=0x7f92f8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 10 monclient: shutdown 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.124+0000 7f931bf96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.125+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 >> v1:172.21.15.117:6789/0 conn(0x7f93140a9d70 legacy=0x7f93141d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.125+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 shutdown_connections 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.125+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 >> 172.21.15.117:0/2314789935 conn(0x7f9314066a00 msgr2=0x7f931413c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.125+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 shutdown_connections 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.126+0000 7f931bf96640 1 -- 172.21.15.117:0/2314789935 wait complete. 2024-03-29T11:27:27.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:27.126+0000 7f931bf96640 1 librados: shutdown 2024-03-29T11:27:27.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:33.940 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:34.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.095+0000 7f0238765640 10 monclient: get_monmap_and_config 2024-03-29T11:27:34.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.095+0000 7f0238765640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.083 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.095+0000 7f0238765640 10 monclient: monmap: 2024-03-29T11:27:34.083 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:34.083 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:34.096327+0000 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:34.096327+0000 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:34.084 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.086 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.096+0000 7f0238765640 1 Processor -- start 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.096+0000 7f0238765640 1 -- start start 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.096+0000 7f0238765640 10 monclient: init 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient(hunting): picked mon.noname-b con 0x7f02300b1f60 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient(hunting): picked mon.noname-c con 0x7f0230071eb0 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient(hunting): picked mon.noname-a con 0x7f0230143d10 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023006ef80 con 0x7f0230143d10 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023006f260 con 0x7f0230071eb0 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02300b40c0 con 0x7f02300b1f60 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.087 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.097+0000 7f0238765640 10 monclient(hunting): authenticate will time out at 1322.406343s 2024-03-29T11:27:34.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f0235d00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0230071eb0 0x7f0230146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:58337/0 (socket says 172.21.15.117:58337) 2024-03-29T11:27:34.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f0235d00640 1 -- 172.21.15.117:0/3272965270 learned_addr learned my addr 172.21.15.117:0/3272965270 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:34.089 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156500544 0 0) 0x7f023006f260 con 0x7f0230071eb0 2024-03-29T11:27:34.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:27:34.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.101+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f021c003520 con 0x7f0230071eb0 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325392369 0 0) 0x7f021c003520 con 0x7f0230071eb0 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 >> v1:172.21.15.117:6790/0 conn(0x7f02300b1f60 legacy=0x7f0230143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 >> v1:172.21.15.117:6789/0 conn(0x7f0230143d10 legacy=0x7f0230146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 monclient: found mon.noname-c 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:27:34.094 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0230155d70 con 0x7f0230071eb0 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f0238765640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f0238765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0228001a10 con 0x7f0230071eb0 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f02354ff640 10 monclient: dump: 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.095 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.102+0000 7f0238765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0228001be0 con 0x7f0230071eb0 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 1 -- 172.21.15.117:0/3272965270 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0228002030 con 0x7f0230071eb0 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f02354ff640 10 monclient: dump: 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.096 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f0238765640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.103+0000 7f0238765640 4 set_mon_vals no callback set 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 10 monclient: shutdown 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 1 -- 172.21.15.117:0/3272965270 >> v1:172.21.15.173:6789/0 conn(0x7f0230071eb0 legacy=0x7f0230146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 1 -- 172.21.15.117:0/3272965270 shutdown_connections 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 1 -- 172.21.15.117:0/3272965270 >> 172.21.15.117:0/3272965270 conn(0x7f0230066a00 msgr2=0x7f0230066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.104+0000 7f0238765640 1 -- 172.21.15.117:0/3272965270 shutdown_connections 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.105+0000 7f0238765640 1 -- 172.21.15.117:0/3272965270 wait complete. 2024-03-29T11:27:34.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.105+0000 7f0238765640 10 monclient: build_initial_monmap 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.105+0000 7f0238765640 10 monclient: monmap: 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 librados: starting msgr at 2024-03-29T11:27:34.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 librados: starting objecter 2024-03-29T11:27:34.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 Processor -- start 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 -- start start 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 librados: setting wanted keys 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 1 librados: calling monclient init 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.106+0000 7f0238765640 10 monclient: init 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient(hunting): picked mon.noname-a con 0x7f02300b1f60 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient(hunting): picked mon.noname-b con 0x7f0230143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient(hunting): picked mon.noname-c con 0x7f02301549f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023013b8d0 con 0x7f02300b1f60 2024-03-29T11:27:34.100 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023014aec0 con 0x7f0230143d10 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023014b1a0 con 0x7f02301549f0 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.109+0000 7f0238765640 10 monclient(hunting): authenticate will time out at 1322.418522s 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0236d02640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f02301549f0 0x7f023013b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:40758/0 (socket says 172.21.15.117:40758) 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0236d02640 1 -- 172.21.15.117:0/2227851263 learned_addr learned my addr 172.21.15.117:0/2227851263 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 client.?.objecter ms_handle_connect 0x7f02301549f0 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046209668 0 0) 0x7f023014b1a0 con 0x7f02301549f0 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:27:34.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.111+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214004eb0 con 0x7f02301549f0 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530526014 0 0) 0x7f0214004eb0 con 0x7f02301549f0 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f023014b1a0 con 0x7f02301549f0 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f02200018f0 con 0x7f02301549f0 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:34.105 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.112+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2867079865 0 0) 0x7f023014b1a0 con 0x7f02301549f0 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 >> v1:172.21.15.173:6789/0 conn(0x7f0230143d10 legacy=0x7f023013aab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 >> v1:172.21.15.117:6789/0 conn(0x7f02300b1f60 legacy=0x7f023013f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:34.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 1 monclient: found mon.c 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02301499c0 con 0x7f02301549f0 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.113+0000 7f0226ffd640 10 client.?.objecter ms_handle_connect 0x7f02300b1f60 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0220001d20 con 0x7f02301549f0 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f0220003a60 con 0x7f02301549f0 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.114+0000 7f0226ffd640 10 monclient: dump: 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:34.107 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0234cfe640 4 set_mon_vals no callback set 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 10 monclient: _renew_subs 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:34.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0230149690 con 0x7f02301549f0 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 10 monclient: _renew_subs 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0230149270 con 0x7f02301549f0 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.115+0000 7f0238765640 1 librados: init done 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.117+0000 7f0238765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.117+0000 7f0238765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.117+0000 7f0238765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:34.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.117+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023014b6c0 con 0x7f02301549f0 2024-03-29T11:27:34.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.120+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f0220003bc0 con 0x7f02301549f0 2024-03-29T11:27:34.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.121+0000 7f0236501640 10 monclient: get_auth_request con 0x7f021406dcf0 auth_method 0 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 10 client.4613.objecter ms_handle_connect 0x7f021406dcf0 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f0220084b20 con 0x7f02301549f0 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 10 client.4613.objecter ms_dispatch 0x7f0230000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:34.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.122+0000 7f0226ffd640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:27:34.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.129+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f023014b6c0 con 0x7f02301549f0 2024-03-29T11:27:34.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.129+0000 7f0226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:34.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.129+0000 7f0226ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:34.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.432+0000 7f0238765640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.432+0000 7f0238765640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.432+0000 7f0238765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:34.426 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:34.426 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.432+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f023014b6c0 con 0x7f02301549f0 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.434+0000 7f0226ffd640 1 -- 172.21.15.117:0/2227851263 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f023014b6c0 con 0x7f02301549f0 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.434+0000 7f0226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.434+0000 7f0226ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 10 librados: watch_flush enter 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 10 librados: watch_flush exit 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 >> v1:172.21.15.117:6816/4276038175 conn(0x7f021406dcf0 legacy=0x7f02140701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 10 monclient: shutdown 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:34.443 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 >> v1:172.21.15.117:6790/0 conn(0x7f02301549f0 legacy=0x7f023013b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:34.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 shutdown_connections 2024-03-29T11:27:34.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 >> 172.21.15.117:0/2227851263 conn(0x7f0230066a00 msgr2=0x7f0230066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:34.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 shutdown_connections 2024-03-29T11:27:34.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 -- 172.21.15.117:0/2227851263 wait complete. 2024-03-29T11:27:34.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:34.438+0000 7f0238765640 1 librados: shutdown 2024-03-29T11:27:34.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:41.266 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.424+0000 7ff82bc16640 10 monclient: get_monmap_and_config 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.424+0000 7ff82bc16640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.424+0000 7ff82bc16640 10 monclient: monmap: 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:41.425864+0000 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:41.425864+0000 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.413 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.414 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:41.414 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:41.414 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:41.414 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.425+0000 7ff82bc16640 1 Processor -- start 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.425+0000 7ff82bc16640 1 -- start start 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.425+0000 7ff82bc16640 10 monclient: init 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-a con 0x7ff82406ade0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-b con 0x7ff82406ed60 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-c con 0x7ff824072340 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82406ff50 con 0x7ff82406ade0 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824070180 con 0x7ff824072340 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82406f310 con 0x7ff82406ed60 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.415 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff82bc16640 10 monclient(hunting): authenticate will time out at 1329.735772s 2024-03-29T11:27:41.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff8291b1640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ff82406ed60 0x7ff824071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:50492/0 (socket says 172.21.15.117:50492) 2024-03-29T11:27:41.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.426+0000 7ff8291b1640 1 -- 172.21.15.117:0/3730712776 learned_addr learned my addr 172.21.15.117:0/3730712776 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:41.416 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052153781 0 0) 0x7ff82406f310 con 0x7ff82406ed60 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff810003520 con 0x7ff82406ed60 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023216955 0 0) 0x7ff824070180 con 0x7ff824072340 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82406f310 con 0x7ff824072340 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981399640 0 0) 0x7ff82406ff50 con 0x7ff82406ade0 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824070180 con 0x7ff82406ade0 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.427+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897635351 0 0) 0x7ff810003520 con 0x7ff82406ed60 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 >> v1:172.21.15.173:6789/0 conn(0x7ff824072340 legacy=0x7ff824151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 >> v1:172.21.15.117:6789/0 conn(0x7ff82406ade0 legacy=0x7ff8241499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 monclient: found mon.noname-b 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240aeed0 con 0x7ff82406ed60 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff82bc16640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff82bc16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff820001690 con 0x7ff82406ed60 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: dump: 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.419 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff82bc16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8200018e0 con 0x7ff82406ed60 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff82bc16640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff82bc16640 4 set_mon_vals no callback set 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 1 -- 172.21.15.117:0/3730712776 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff820001c70 con 0x7ff82406ed60 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.428+0000 7ff8289b0640 10 monclient: dump: 2024-03-29T11:27:41.420 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:41.421 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.426 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.429+0000 7ff82bc16640 10 monclient: shutdown 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.429+0000 7ff82bc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.429+0000 7ff82bc16640 1 -- 172.21.15.117:0/3730712776 >> v1:172.21.15.117:6790/0 conn(0x7ff82406ed60 legacy=0x7ff824071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.429+0000 7ff82bc16640 1 -- 172.21.15.117:0/3730712776 shutdown_connections 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.429+0000 7ff82bc16640 1 -- 172.21.15.117:0/3730712776 >> 172.21.15.117:0/3730712776 conn(0x7ff8240b4930 msgr2=0x7ff8240b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.430+0000 7ff82bc16640 1 -- 172.21.15.117:0/3730712776 shutdown_connections 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.430+0000 7ff82bc16640 1 -- 172.21.15.117:0/3730712776 wait complete. 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.430+0000 7ff82bc16640 10 monclient: build_initial_monmap 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.430+0000 7ff82bc16640 10 monclient: monmap: 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:41.427 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 librados: starting msgr at 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 librados: starting objecter 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 Processor -- start 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 -- start start 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 librados: setting wanted keys 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 1 librados: calling monclient init 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.431+0000 7ff82bc16640 10 monclient: init 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:41.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-a con 0x7ff82406ade0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-b con 0x7ff82406ed60 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient(hunting): picked mon.noname-c con 0x7ff824072340 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8241db130 con 0x7ff82406ade0 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240ab820 con 0x7ff82406ed60 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8241dbfb0 con 0x7ff824072340 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff82bc16640 10 monclient(hunting): authenticate will time out at 1329.741381s 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff8291b1640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff82406ed60 0x7ff8241db8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:33244/0 (socket says 172.21.15.117:33244) 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff8291b1640 1 -- 172.21.15.117:0/2371220389 learned_addr learned my addr 172.21.15.117:0/2371220389 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff82406ed60 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.432+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff824072340 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623547079 0 0) 0x7ff8240ab820 con 0x7ff82406ed60 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff82406ed60 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff82406ade0 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048680307 0 0) 0x7ff8241db130 con 0x7ff82406ade0 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.433+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8240ab820 con 0x7ff82406ade0 2024-03-29T11:27:41.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885822028 0 0) 0x7ff8241dbfb0 con 0x7ff824072340 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8241db130 con 0x7ff824072340 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775246348 0 0) 0x7ff808004eb0 con 0x7ff82406ed60 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8241dbfb0 con 0x7ff82406ed60 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff820001bf0 con 0x7ff82406ed60 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.431 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035852765 0 0) 0x7ff8240ab820 con 0x7ff82406ade0 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff82406ade0 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.434+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff81c001cb0 con 0x7ff82406ade0 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:41.432 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3743133688 0 0) 0x7ff8241dbfb0 con 0x7ff82406ed60 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 >> v1:172.21.15.117:6790/0 conn(0x7ff824072340 legacy=0x7ff8241e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 >> v1:172.21.15.117:6789/0 conn(0x7ff82406ade0 legacy=0x7ff8240a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 monclient: found mon.b 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:41.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8241db380 con 0x7ff82406ed60 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 10 monclient: _renew_subs 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff824144a20 con 0x7ff82406ed60 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 10 monclient: _renew_subs 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8240a9f60 con 0x7ff82406ed60 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff82bc16640 1 librados: init done 2024-03-29T11:27:41.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff820001d50 con 0x7ff82406ed60 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.435+0000 7ff81bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.437+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ff8200010e0 con 0x7ff82406ed60 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff81a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff81a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff82bc16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff81a7fc640 10 monclient: dump: 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:41.435 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7ff82001acc0 con 0x7ff82406ed60 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff82bc16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff82bc16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:41.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.438+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c005080 con 0x7ff82406ed60 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff8299b2640 10 monclient: get_auth_request con 0x7ff80806ddd0 auth_method 0 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7ff820084b20 con 0x7ff82406ed60 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 10 client.4516.objecter ms_dispatch 0x7ff824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.439+0000 7ff81a7fc640 10 client.4516.objecter ms_handle_connect 0x7ff80806ddd0 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.446+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff820050400 con 0x7ff82406ed60 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.446+0000 7ff81a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:41.437 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.446+0000 7ff81a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:41.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.810+0000 7ff82bc16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:41.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.810+0000 7ff82bc16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:41.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.810+0000 7ff82bc16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:27:41.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.810+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff80c005600 con 0x7ff82406ed60 2024-03-29T11:27:41.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.811+0000 7ff81a7fc640 1 -- 172.21.15.117:0/2371220389 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7ff820052330 con 0x7ff82406ed60 2024-03-29T11:27:41.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.811+0000 7ff81a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.811+0000 7ff81a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:41.800 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 10 librados: watch_flush enter 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 10 librados: watch_flush exit 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:27:41.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 >> v1:172.21.15.117:6816/4276038175 conn(0x7ff80806ddd0 legacy=0x7ff808070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 10 monclient: shutdown 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:41.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 >> v1:172.21.15.173:6789/0 conn(0x7ff82406ed60 legacy=0x7ff8241db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 shutdown_connections 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 >> 172.21.15.117:0/2371220389 conn(0x7ff8240b4930 msgr2=0x7ff8240b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.814+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 shutdown_connections 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.815+0000 7ff82bc16640 1 -- 172.21.15.117:0/2371220389 wait complete. 2024-03-29T11:27:41.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:41.815+0000 7ff82bc16640 1 librados: shutdown 2024-03-29T11:27:41.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:48.625 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.782+0000 7f848af79640 10 monclient: get_monmap_and_config 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.782+0000 7f848af79640 10 monclient: build_initial_monmap 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.782+0000 7f848af79640 10 monclient: monmap: 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:48.783538+0000 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:48.783538+0000 2024-03-29T11:27:48.771 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:48.772 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.772 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:48.772 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:48.772 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:48.772 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.774 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.786+0000 7f848af79640 1 Processor -- start 2024-03-29T11:27:48.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.786+0000 7f848af79640 1 -- start start 2024-03-29T11:27:48.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.786+0000 7f848af79640 10 monclient: init 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient(hunting): picked mon.noname-b con 0x7f84841422b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient(hunting): picked mon.noname-c con 0x7f8484142690 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient(hunting): picked mon.noname-a con 0x7f84840b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8484142c80 con 0x7f84840b17d0 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840aae40 con 0x7f8484142690 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848413fbd0 con 0x7f84841422b0 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.787+0000 7f848af79640 10 monclient(hunting): authenticate will time out at 1337.096444s 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.789+0000 7f8483fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f8484142690 0x7f84840b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:53181/0 (socket says 172.21.15.117:53181) 2024-03-29T11:27:48.779 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.789+0000 7f8483fff640 1 -- 172.21.15.117:0/270697176 learned_addr learned my addr 172.21.15.117:0/270697176 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.790+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452600710 0 0) 0x7f84840aae40 con 0x7f8484142690 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f846c003520 con 0x7f8484142690 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439100676 0 0) 0x7f8484142c80 con 0x7f84840b17d0 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:48.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84840aae40 con 0x7f84840b17d0 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509000742 0 0) 0x7f848413fbd0 con 0x7f84841422b0 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484142c80 con 0x7f84841422b0 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106033969 0 0) 0x7f8484142c80 con 0x7f84841422b0 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 >> v1:172.21.15.173:6789/0 conn(0x7f8484142690 legacy=0x7f84840b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 >> v1:172.21.15.117:6789/0 conn(0x7f84840b17d0 legacy=0x7f84840b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:48.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484151940 con 0x7f84841422b0 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f848af79640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f848af79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f84740015b0 con 0x7f84841422b0 2024-03-29T11:27:48.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: dump: 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f848af79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84740017d0 con 0x7f84841422b0 2024-03-29T11:27:48.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:48.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f84837fe640 1 -- 172.21.15.117:0/270697176 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8474001b60 con 0x7f84841422b0 2024-03-29T11:27:48.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f848af79640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:48.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.791+0000 7f848af79640 4 set_mon_vals no callback set 2024-03-29T11:27:48.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f84837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f84837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f84837fe640 10 monclient: dump: 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:48.786 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f848af79640 10 monclient: shutdown 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f848af79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:48.787 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.792+0000 7f848af79640 1 -- 172.21.15.117:0/270697176 >> v1:172.21.15.117:6790/0 conn(0x7f84841422b0 legacy=0x7f848413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.793+0000 7f848af79640 1 -- 172.21.15.117:0/270697176 shutdown_connections 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.793+0000 7f848af79640 1 -- 172.21.15.117:0/270697176 >> 172.21.15.117:0/270697176 conn(0x7f8484066a00 msgr2=0x7f8484066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.793+0000 7f848af79640 1 -- 172.21.15.117:0/270697176 shutdown_connections 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.794+0000 7f848af79640 1 -- 172.21.15.117:0/270697176 wait complete. 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.794+0000 7f848af79640 10 monclient: build_initial_monmap 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.794+0000 7f848af79640 10 monclient: monmap: 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:48.790 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 librados: starting msgr at 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 librados: starting objecter 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 Processor -- start 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 -- start start 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 librados: setting wanted keys 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 1 librados: calling monclient init 2024-03-29T11:27:48.791 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.795+0000 7f848af79640 10 monclient: init 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient(hunting): picked mon.noname-a con 0x7f84840b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient(hunting): picked mon.noname-b con 0x7f84841422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient(hunting): picked mon.noname-c con 0x7f8484142690 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84841da900 con 0x7f84840b17d0 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840b03a0 con 0x7f84841422b0 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840b1280 con 0x7f8484142690 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.796+0000 7f848af79640 10 monclient(hunting): authenticate will time out at 1337.105725s 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.799+0000 7f8489516640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f8484142690 0x7f84841e1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:37790/0 (socket says 172.21.15.117:37790) 2024-03-29T11:27:48.792 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.799+0000 7f8489516640 1 -- 172.21.15.117:0/308561609 learned_addr learned my addr 172.21.15.117:0/308561609 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.800+0000 7f84817fa640 10 client.?.objecter ms_handle_connect 0x7f8484142690 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.800+0000 7f84817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578025485 0 0) 0x7f84840b1280 con 0x7f8484142690 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:27:48.793 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8470004eb0 con 0x7f8484142690 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81092644 0 0) 0x7f8470004eb0 con 0x7f8484142690 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84840b1280 con 0x7f8484142690 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 client.?.objecter ms_handle_connect 0x7f84840b17d0 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f847c001d10 con 0x7f8484142690 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): dump: 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.794 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 874144096 0 0) 0x7f84840b1280 con 0x7f8484142690 2024-03-29T11:27:48.795 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.801+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 >> v1:172.21.15.173:6789/0 conn(0x7f84841422b0 legacy=0x7f84841dad90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.802+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 >> v1:172.21.15.117:6789/0 conn(0x7f84840b17d0 legacy=0x7f84840af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 1 monclient: found mon.c 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840ac4d0 con 0x7f8484142690 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f84817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:48.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 10 monclient: _renew_subs 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84840afe90 con 0x7f8484142690 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 10 monclient: _renew_subs 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841db680 con 0x7f8484142690 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.803+0000 7f848af79640 1 librados: init done 2024-03-29T11:27:48.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f847c002010 con 0x7f8484142690 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f847c003de0 con 0x7f8484142690 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.804+0000 7f84817fa640 10 monclient: dump: 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:48.801 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.805+0000 7f8482ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f847c003f40 con 0x7f8484142690 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f847c084e50 con 0x7f8484142690 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 10 client.4625.objecter ms_dispatch 0x7f8484000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.807+0000 7f84817fa640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f8488d15640 10 monclient: get_auth_request con 0x7f847006db60 auth_method 0 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f848af79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f848af79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f848af79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464005080 con 0x7f8484142690 2024-03-29T11:27:48.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.808+0000 7f84817fa640 10 client.4625.objecter ms_handle_connect 0x7f847006db60 2024-03-29T11:27:48.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.816+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f847c050730 con 0x7f8484142690 2024-03-29T11:27:48.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.816+0000 7f84817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:48.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:48.816+0000 7f84817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:49.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.154+0000 7f848af79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.154+0000 7f848af79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.154+0000 7f848af79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:27:49.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.154+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8464005330 con 0x7f8484142690 2024-03-29T11:27:49.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.155+0000 7f84817fa640 1 -- 172.21.15.117:0/308561609 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f847c052660 con 0x7f8484142690 2024-03-29T11:27:49.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.155+0000 7f84817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:49.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.155+0000 7f84817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:49.144 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:49.144 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:49.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.158+0000 7f848af79640 10 librados: watch_flush enter 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.158+0000 7f848af79640 10 librados: watch_flush exit 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.158+0000 7f848af79640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.158+0000 7f848af79640 10 client.4625.objecter successfully canceled tick 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 >> v1:172.21.15.117:6816/4276038175 conn(0x7f847006db60 legacy=0x7f8470070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 10 monclient: shutdown 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 >> v1:172.21.15.117:6790/0 conn(0x7f8484142690 legacy=0x7f84841e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 shutdown_connections 2024-03-29T11:27:49.148 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 >> 172.21.15.117:0/308561609 conn(0x7f8484066a00 msgr2=0x7f8484066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:49.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 shutdown_connections 2024-03-29T11:27:49.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.159+0000 7f848af79640 1 -- 172.21.15.117:0/308561609 wait complete. 2024-03-29T11:27:49.149 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:49.160+0000 7f848af79640 1 librados: shutdown 2024-03-29T11:27:49.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:55.971 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.119+0000 7ffa5464a640 10 monclient: get_monmap_and_config 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.119+0000 7ffa5464a640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.119+0000 7ffa5464a640 10 monclient: monmap: 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:27:56.120171+0000 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:27:56.120171+0000 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:27:56.108 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:27:56.109 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.120+0000 7ffa5464a640 1 Processor -- start 2024-03-29T11:27:56.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.120+0000 7ffa5464a640 1 -- start start 2024-03-29T11:27:56.109 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.120+0000 7ffa5464a640 10 monclient: init 2024-03-29T11:27:56.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.121+0000 7ffa5464a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.121+0000 7ffa5464a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:56.110 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.121+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c1422b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:56.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c142690 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:56.111 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa523e6640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7ffa4c1422b0 0x7ffa4c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:48214/0 (socket says 172.21.15.117:48214) 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c0b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c142c80 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0aae40 con 0x7ffa4c142690 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c13fbd0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.122+0000 7ffa5464a640 10 monclient(hunting): authenticate will time out at 1344.431553s 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.124+0000 7ffa523e6640 1 -- 172.21.15.117:0/4032490481 learned_addr learned my addr 172.21.15.117:0/4032490481 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913274462 0 0) 0x7ffa4c13fbd0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa38003520 con 0x7ffa4c1422b0 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 284646672 0 0) 0x7ffa38003520 con 0x7ffa4c1422b0 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 >> v1:172.21.15.173:6789/0 conn(0x7ffa4c142690 legacy=0x7ffa4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:56.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.125+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 >> v1:172.21.15.117:6789/0 conn(0x7ffa4c0b17d0 legacy=0x7ffa4c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 1 monclient: found mon.noname-b 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c151940 con 0x7ffa4c1422b0 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa440019f0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa513e4640 10 monclient: dump: 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.119 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa5464a640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.126+0000 7ffa5464a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa44001c50 con 0x7ffa4c1422b0 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 1 -- 172.21.15.117:0/4032490481 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa44002020 con 0x7ffa4c1422b0 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa513e4640 10 monclient: dump: 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.120 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa5464a640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.127+0000 7ffa5464a640 4 set_mon_vals no callback set 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 10 monclient: shutdown 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 1 -- 172.21.15.117:0/4032490481 >> v1:172.21.15.117:6790/0 conn(0x7ffa4c1422b0 legacy=0x7ffa4c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 1 -- 172.21.15.117:0/4032490481 shutdown_connections 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 1 -- 172.21.15.117:0/4032490481 >> 172.21.15.117:0/4032490481 conn(0x7ffa4c066a00 msgr2=0x7ffa4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.128+0000 7ffa5464a640 1 -- 172.21.15.117:0/4032490481 shutdown_connections 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.129+0000 7ffa5464a640 1 -- 172.21.15.117:0/4032490481 wait complete. 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.129+0000 7ffa5464a640 10 monclient: build_initial_monmap 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.129+0000 7ffa5464a640 10 monclient: monmap: 2024-03-29T11:27:56.121 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:27:56.122 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.130+0000 7ffa5464a640 1 librados: starting msgr at 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.130+0000 7ffa5464a640 1 librados: starting objecter 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.130+0000 7ffa5464a640 1 Processor -- start 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.130+0000 7ffa5464a640 1 -- start start 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.130+0000 7ffa5464a640 1 librados: setting wanted keys 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.131+0000 7ffa5464a640 1 librados: calling monclient init 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.131+0000 7ffa5464a640 10 monclient: init 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.131+0000 7ffa5464a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:56.131 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.131+0000 7ffa5464a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.131+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4c0b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4c1422b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4c142690 addr v1:172.21.15.117:6789/0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1320a0 con 0x7ffa4c142690 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c131d10 con 0x7ffa4c1422b0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c06eb40 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.132+0000 7ffa5464a640 10 monclient(hunting): authenticate will time out at 1344.441531s 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.133+0000 7ffa51be5640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ffa4c1422b0 0x7ffa4c132530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:36044/0 (socket says 172.21.15.117:36044) 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.133+0000 7ffa51be5640 1 -- 172.21.15.117:0/213818370 learned_addr learned my addr 172.21.15.117:0/213818370 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.133+0000 7ffa42ffd640 10 client.?.objecter ms_handle_connect 0x7ffa4c0b17d0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.133+0000 7ffa42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.134+0000 7ffa42ffd640 10 client.?.objecter ms_handle_connect 0x7ffa4c1422b0 2024-03-29T11:27:56.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.134+0000 7ffa42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.134+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459885354 0 0) 0x7ffa4c131d10 con 0x7ffa4c1422b0 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.135+0000 7ffa42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.135+0000 7ffa42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.135+0000 7ffa42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.135+0000 7ffa42ffd640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593384682 0 0) 0x7ffa4c06eb40 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c131d10 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 client.?.objecter ms_handle_connect 0x7ffa4c142690 2024-03-29T11:27:56.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.136+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234035170 0 0) 0x7ffa4c1320a0 con 0x7ffa4c142690 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c06eb40 con 0x7ffa4c142690 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3339242521 0 0) 0x7ffa30004eb0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c1320a0 con 0x7ffa4c1422b0 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa48001750 con 0x7ffa4c1422b0 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.134 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405060045 0 0) 0x7ffa4c06eb40 con 0x7ffa4c142690 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa30004eb0 con 0x7ffa4c142690 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1692282820 0 0) 0x7ffa4c131d10 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa4c06eb40 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.135 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa3c0016f0 con 0x7ffa4c142690 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa44001f60 con 0x7ffa4c0b17d0 2024-03-29T11:27:56.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.137+0000 7ffa42ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3843147561 0 0) 0x7ffa30004eb0 con 0x7ffa4c142690 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 >> v1:172.21.15.117:6790/0 conn(0x7ffa4c0b17d0 legacy=0x7ffa4c12fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 >> v1:172.21.15.173:6789/0 conn(0x7ffa4c1422b0 legacy=0x7ffa4c132530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 1 monclient: found mon.a 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0b6e60 con 0x7ffa4c142690 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 10 monclient: _renew_subs 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c0b70b0 con 0x7ffa4c142690 2024-03-29T11:27:56.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 10 monclient: _renew_subs 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c071640 con 0x7ffa4c142690 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.138+0000 7ffa5464a640 1 librados: init done 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.139+0000 7ffa5464a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.139+0000 7ffa5464a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.139+0000 7ffa5464a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.139+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c06f060 con 0x7ffa4c142690 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa3c001850 con 0x7ffa4c142690 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7ffa3c001d80 con 0x7ffa4c142690 2024-03-29T11:27:56.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa50be3640 4 set_mon_vals no callback set 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.140+0000 7ffa42ffd640 10 monclient: dump: 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:27:56.140 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.142+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7ffa3c019460 con 0x7ffa4c142690 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.143+0000 7ffa523e6640 10 monclient: get_auth_request con 0x7ffa3006e130 auth_method 0 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7ffa3c083400 con 0x7ffa4c142690 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 10 client.4524.objecter ms_dispatch 0x7ffa4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.144+0000 7ffa42ffd640 10 client.4524.objecter ms_handle_connect 0x7ffa3006e130 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.148+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa3c04eb10 con 0x7ffa4c142690 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.148+0000 7ffa42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:56.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.148+0000 7ffa42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:56.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.467+0000 7ffa5464a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.467+0000 7ffa5464a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.467+0000 7ffa5464a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:27:56.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.467+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c0720a0 con 0x7ffa4c142690 2024-03-29T11:27:56.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.471+0000 7ffa42ffd640 1 -- 172.21.15.117:0/213818370 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7ffa3c050c10 con 0x7ffa4c142690 2024-03-29T11:27:56.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.471+0000 7ffa42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:56.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.471+0000 7ffa42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:56.461 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:27:56.461 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:56.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 10 librados: watch_flush enter 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 10 librados: watch_flush exit 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 >> v1:172.21.15.117:6816/4276038175 conn(0x7ffa3006e130 legacy=0x7ffa300705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 10 monclient: shutdown 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.475+0000 7ffa5464a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:56.465 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.476+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 >> v1:172.21.15.117:6789/0 conn(0x7ffa4c142690 legacy=0x7ffa4c131600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:56.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.478+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 shutdown_connections 2024-03-29T11:27:56.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.478+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 >> 172.21.15.117:0/213818370 conn(0x7ffa4c066a00 msgr2=0x7ffa4c13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:56.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.479+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 shutdown_connections 2024-03-29T11:27:56.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.479+0000 7ffa5464a640 1 -- 172.21.15.117:0/213818370 wait complete. 2024-03-29T11:27:56.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:27:56.479+0000 7ffa5464a640 1 librados: shutdown 2024-03-29T11:27:56.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:03.296 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:03.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.458+0000 7f88343b5640 10 monclient: get_monmap_and_config 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.458+0000 7f88343b5640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.458+0000 7f88343b5640 10 monclient: monmap: 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:03.459880+0000 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:03.459880+0000 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:03.448 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.460+0000 7f88343b5640 1 Processor -- start 2024-03-29T11:28:03.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.460+0000 7f88343b5640 1 -- start start 2024-03-29T11:28:03.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.460+0000 7f88343b5640 10 monclient: init 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-a con 0x7f882c06ee40 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-c con 0x7f882c0aeda0 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-b con 0x7f882c13c470 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c06fdb0 con 0x7f882c06ee40 2024-03-29T11:28:03.453 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c071090 con 0x7f882c0aeda0 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c071370 con 0x7f882c13c470 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f88343b5640 10 monclient(hunting): authenticate will time out at 1351.770199s 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f8831950640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f882c0aeda0 0x7f882c071790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:50128/0 (socket says 172.21.15.117:50128) 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f8831950640 1 -- 172.21.15.117:0/1695522975 learned_addr learned my addr 172.21.15.117:0/1695522975 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153598395 0 0) 0x7f882c071090 con 0x7f882c0aeda0 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.461+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8818003520 con 0x7f882c0aeda0 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.462+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895654966 0 0) 0x7f8818003520 con 0x7f882c0aeda0 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.462+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 >> v1:172.21.15.117:6790/0 conn(0x7f882c13c470 legacy=0x7f882c06ab70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:03.454 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.463+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 >> v1:172.21.15.117:6789/0 conn(0x7f882c06ee40 legacy=0x7f882c13bd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 1 monclient: found mon.noname-c 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c070220 con 0x7f882c0aeda0 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8824001690 con 0x7f882c0aeda0 2024-03-29T11:28:03.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:03.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f883114f640 10 monclient: dump: 2024-03-29T11:28:03.457 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f88343b5640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:28:03.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.465+0000 7f88343b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8824001860 con 0x7f882c0aeda0 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 1 -- 172.21.15.117:0/1695522975 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8824001cb0 con 0x7f882c0aeda0 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f883114f640 10 monclient: dump: 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.460 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f88343b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.467+0000 7f88343b5640 4 set_mon_vals no callback set 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.468+0000 7f88343b5640 10 monclient: shutdown 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.468+0000 7f88343b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.468+0000 7f88343b5640 1 -- 172.21.15.117:0/1695522975 >> v1:172.21.15.173:6789/0 conn(0x7f882c0aeda0 legacy=0x7f882c071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.461 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.468+0000 7f88343b5640 1 -- 172.21.15.117:0/1695522975 shutdown_connections 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.468+0000 7f88343b5640 1 -- 172.21.15.117:0/1695522975 >> 172.21.15.117:0/1695522975 conn(0x7f882c067470 msgr2=0x7f882c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.469+0000 7f88343b5640 1 -- 172.21.15.117:0/1695522975 shutdown_connections 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.469+0000 7f88343b5640 1 -- 172.21.15.117:0/1695522975 wait complete. 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 10 monclient: build_initial_monmap 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 10 monclient: monmap: 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:03.462 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 librados: starting msgr at 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 librados: starting objecter 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 Processor -- start 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 -- start start 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 librados: setting wanted keys 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 1 librados: calling monclient init 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.470+0000 7f88343b5640 10 monclient: init 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:03.463 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-c con 0x7f882c06ee40 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-b con 0x7f882c0aeda0 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient(hunting): picked mon.noname-a con 0x7f882c13c470 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0ab4d0 con 0x7f882c13c470 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0ab7b0 con 0x7f882c0aeda0 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0aa1c0 con 0x7f882c06ee40 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:03.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.471+0000 7f88343b5640 10 monclient(hunting): authenticate will time out at 1351.780461s 2024-03-29T11:28:03.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8831950640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f882c0aeda0 0x7f882c141180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:57810/0 (socket says 172.21.15.117:57810) 2024-03-29T11:28:03.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8831950640 1 -- 172.21.15.117:0/553842583 learned_addr learned my addr 172.21.15.117:0/553842583 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:03.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 client.?.objecter ms_handle_connect 0x7f882c0aeda0 2024-03-29T11:28:03.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518289354 0 0) 0x7f882c0ab7b0 con 0x7f882c0aeda0 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.475+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8810004eb0 con 0x7f882c0aeda0 2024-03-29T11:28:03.468 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.476+0000 7f8822ffd640 10 client.?.objecter ms_handle_connect 0x7f882c13c470 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.476+0000 7f8822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293909179 0 0) 0x7f8810004eb0 con 0x7f882c0aeda0 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f882c0ab7b0 con 0x7f882c0aeda0 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165454830 0 0) 0x7f882c0ab4d0 con 0x7f882c13c470 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8810004eb0 con 0x7f882c13c470 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 client.?.objecter ms_handle_connect 0x7f882c06ee40 2024-03-29T11:28:03.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8824001b70 con 0x7f882c0aeda0 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.470 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198685531 0 0) 0x7f882c0aa1c0 con 0x7f882c06ee40 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882c0ab4d0 con 0x7f882c06ee40 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515111782 0 0) 0x7f8810004eb0 con 0x7f882c13c470 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f882c0aa1c0 con 0x7f882c13c470 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8828001990 con 0x7f882c13c470 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:03.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.477+0000 7f8822ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.472 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783871541 0 0) 0x7f882c0ab4d0 con 0x7f882c06ee40 2024-03-29T11:28:03.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8810004eb0 con 0x7f882c06ee40 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f881c001510 con 0x7f882c06ee40 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.479+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1852372220 0 0) 0x7f882c0ab7b0 con 0x7f882c0aeda0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 >> v1:172.21.15.117:6790/0 conn(0x7f882c06ee40 legacy=0x7f882c0a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 >> v1:172.21.15.117:6789/0 conn(0x7f882c13c470 legacy=0x7f882c141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 1 monclient: found mon.b 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c0a9e60 con 0x7f882c0aeda0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f8822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.480+0000 7f88343b5640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 10 monclient: _renew_subs 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8824004bf0 con 0x7f882c0aeda0 2024-03-29T11:28:03.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c12d330 con 0x7f882c0aeda0 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 10 monclient: _renew_subs 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c150a70 con 0x7f882c0aeda0 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f88343b5640 1 librados: init done 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8824004f00 con 0x7f882c0aeda0 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8822ffd640 10 monclient: dump: 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.479 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.481+0000 7f8823fff640 4 set_mon_vals no callback set 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.482+0000 7f88343b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.482+0000 7f88343b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.482+0000 7f88343b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.483+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8814005080 con 0x7f882c0aeda0 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.483+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f882401aed0 con 0x7f882c0aeda0 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.484+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f8824084c70 con 0x7f882c0aeda0 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.484+0000 7f8822ffd640 10 client.4528.objecter ms_dispatch 0x7f882c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.484+0000 7f8832151640 10 monclient: get_auth_request con 0x7f88100768d0 auth_method 0 2024-03-29T11:28:03.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.485+0000 7f8822ffd640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.485+0000 7f8822ffd640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.485+0000 7f8822ffd640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.485+0000 7f8822ffd640 10 client.4528.objecter ms_handle_connect 0x7f88100768d0 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.491+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8824050550 con 0x7f882c0aeda0 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.491+0000 7f8822ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:03.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.491+0000 7f8822ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:03.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.813+0000 7f88343b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:03.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.813+0000 7f88343b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:03.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.814+0000 7f88343b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:03.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.814+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8814005330 con 0x7f882c0aeda0 2024-03-29T11:28:03.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.816+0000 7f8822ffd640 1 -- 172.21.15.117:0/553842583 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f8824052480 con 0x7f882c0aeda0 2024-03-29T11:28:03.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.816+0000 7f8822ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:03.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.816+0000 7f8822ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:03.805 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:03.805 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:03.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.819+0000 7f88343b5640 10 librados: watch_flush enter 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.819+0000 7f88343b5640 10 librados: watch_flush exit 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.819+0000 7f88343b5640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.819+0000 7f88343b5640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.820+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 >> v1:172.21.15.117:6816/4276038175 conn(0x7f88100768d0 legacy=0x7f8810078d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.820+0000 7f88343b5640 10 monclient: shutdown 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.820+0000 7f88343b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:03.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.820+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 >> v1:172.21.15.173:6789/0 conn(0x7f882c0aeda0 legacy=0x7f882c141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:03.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.822+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 shutdown_connections 2024-03-29T11:28:03.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.822+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 >> 172.21.15.117:0/553842583 conn(0x7f882c067470 msgr2=0x7f882c0afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:03.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.822+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 shutdown_connections 2024-03-29T11:28:03.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.822+0000 7f88343b5640 1 -- 172.21.15.117:0/553842583 wait complete. 2024-03-29T11:28:03.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:03.822+0000 7f88343b5640 1 librados: shutdown 2024-03-29T11:28:03.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:10.637 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:10.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.786+0000 7fb37f3d6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:10.775 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.786+0000 7fb37f3d6640 10 monclient: build_initial_monmap 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.786+0000 7fb37f3d6640 10 monclient: monmap: 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:10.787162+0000 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:10.787162+0000 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:10.776 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.787+0000 7fb37f3d6640 1 Processor -- start 2024-03-29T11:28:10.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.787+0000 7fb37f3d6640 1 -- start start 2024-03-29T11:28:10.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.787+0000 7fb37f3d6640 10 monclient: init 2024-03-29T11:28:10.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:10.780 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3781457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb37813f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3780a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3780aaef0 con 0x7fb37813f650 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb378145bd0 con 0x7fb3781457f0 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb378145eb0 con 0x7fb3780a9d70 2024-03-29T11:28:10.781 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37f3d6640 10 monclient(hunting): authenticate will time out at 1359.097252s 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37d973640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb3780a9d70 0x7fb37814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44716/0 (socket says 172.21.15.117:44716) 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb37d973640 1 -- 172.21.15.117:0/1748376127 learned_addr learned my addr 172.21.15.117:0/1748376127 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975997634 0 0) 0x7fb378145eb0 con 0x7fb3780a9d70 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:28:10.782 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb364003520 con 0x7fb3780a9d70 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536993826 0 0) 0x7fb364003520 con 0x7fb3780a9d70 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 >> v1:172.21.15.173:6789/0 conn(0x7fb3781457f0 legacy=0x7fb37813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.788+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 >> v1:172.21.15.117:6789/0 conn(0x7fb37813f650 legacy=0x7fb3780b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 1 monclient: found mon.noname-b 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378146340 con 0x7fb3780a9d70 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb368001690 con 0x7fb3780a9d70 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:10.783 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb36ffff640 10 monclient: dump: 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.784 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:10.785 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:10.785 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:10.785 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb37f3d6640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:28:10.785 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.790+0000 7fb37f3d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:10.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb368001860 con 0x7fb3780a9d70 2024-03-29T11:28:10.797 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 1 -- 172.21.15.117:0/1748376127 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb368001cb0 con 0x7fb3780a9d70 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb36ffff640 10 monclient: dump: 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:10.798 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb37f3d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.791+0000 7fb37f3d6640 4 set_mon_vals no callback set 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 10 monclient: shutdown 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:10.799 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 1 -- 172.21.15.117:0/1748376127 >> v1:172.21.15.117:6790/0 conn(0x7fb3780a9d70 legacy=0x7fb37814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 1 -- 172.21.15.117:0/1748376127 shutdown_connections 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 1 -- 172.21.15.117:0/1748376127 >> 172.21.15.117:0/1748376127 conn(0x7fb378066a00 msgr2=0x7fb378066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 1 -- 172.21.15.117:0/1748376127 shutdown_connections 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.792+0000 7fb37f3d6640 1 -- 172.21.15.117:0/1748376127 wait complete. 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.793+0000 7fb37f3d6640 10 monclient: build_initial_monmap 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.793+0000 7fb37f3d6640 10 monclient: monmap: 2024-03-29T11:28:10.800 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 librados: starting msgr at 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 librados: starting objecter 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 Processor -- start 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 -- start start 2024-03-29T11:28:10.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 librados: setting wanted keys 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 1 librados: calling monclient init 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.794+0000 7fb37f3d6640 10 monclient: init 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3780a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37d172640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fb3780a9d70 0x7fb3781deea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46864/0 (socket says 172.21.15.117:46864) 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37d172640 1 -- 172.21.15.117:0/2681624440 learned_addr learned my addr 172.21.15.117:0/2681624440 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb37813f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3781457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.795+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37806efa0 con 0x7fb3781457f0 2024-03-29T11:28:10.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37806f210 con 0x7fb37813f650 2024-03-29T11:28:10.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb37806f4f0 con 0x7fb3780a9d70 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb37f3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb37f3d6640 10 monclient(hunting): authenticate will time out at 1359.105224s 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 client.?.objecter ms_handle_connect 0x7fb3780a9d70 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 client.?.objecter ms_handle_connect 0x7fb37813f650 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147953628 0 0) 0x7fb37806f4f0 con 0x7fb3780a9d70 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:10.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35c004eb0 con 0x7fb3780a9d70 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433448896 0 0) 0x7fb37806f210 con 0x7fb37813f650 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:28:10.805 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb37806f4f0 con 0x7fb37813f650 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745515793 0 0) 0x7fb35c004eb0 con 0x7fb3780a9d70 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb35c0054e0 con 0x7fb3780a9d70 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb370001d10 con 0x7fb3780a9d70 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): dump: 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:10.806 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.796+0000 7fb36dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.797+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2092792703 0 0) 0x7fb35c0054e0 con 0x7fb3780a9d70 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.797+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 >> v1:172.21.15.173:6789/0 conn(0x7fb37813f650 legacy=0x7fb3781df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.797+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 >> v1:172.21.15.117:6789/0 conn(0x7fb3781457f0 legacy=0x7fb3781e6170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 1 monclient: found mon.c 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37806df30 con 0x7fb3780a9d70 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:10.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb36dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 10 monclient: _renew_subs 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37813fda0 con 0x7fb3780a9d70 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 10 monclient: _renew_subs 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb378142ad0 con 0x7fb3780a9d70 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.798+0000 7fb37f3d6640 1 librados: init done 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.800+0000 7fb37f3d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.800+0000 7fb37f3d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.800+0000 7fb37f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.800+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37806e170 con 0x7fb3780a9d70 2024-03-29T11:28:10.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb370003990 con 0x7fb3780a9d70 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fb370003d60 con 0x7fb3780a9d70 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 10 monclient: dump: 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:10.809 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36f7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.801+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fb370003f80 con 0x7fb3780a9d70 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fb370085f50 con 0x7fb3780a9d70 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb36dffb640 10 client.4640.objecter ms_dispatch 0x7fb378000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb36dffb640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb36dffb640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb36dffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.802+0000 7fb37c971640 10 monclient: get_auth_request con 0x7fb35c06dff0 auth_method 0 2024-03-29T11:28:10.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.803+0000 7fb36dffb640 10 client.4640.objecter ms_handle_connect 0x7fb35c06dff0 2024-03-29T11:28:10.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.809+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb37806e170 con 0x7fb3780a9d70 2024-03-29T11:28:10.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.809+0000 7fb36dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:10.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:10.809+0000 7fb36dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:11.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb37f3d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb37f3d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb37f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:11.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb37806e170 con 0x7fb3780a9d70 2024-03-29T11:28:11.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb36dffb640 1 -- 172.21.15.117:0/2681624440 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fb37806e170 con 0x7fb3780a9d70 2024-03-29T11:28:11.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb36dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:11.116 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.126+0000 7fb36dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:11.117 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:11.117 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:11.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.131+0000 7fb37f3d6640 10 librados: watch_flush enter 2024-03-29T11:28:11.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.131+0000 7fb37f3d6640 10 librados: watch_flush exit 2024-03-29T11:28:11.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.131+0000 7fb37f3d6640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:28:11.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.131+0000 7fb37f3d6640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:28:11.122 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 >> v1:172.21.15.117:6816/4276038175 conn(0x7fb35c06dff0 legacy=0x7fb35c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 10 monclient: shutdown 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 >> v1:172.21.15.117:6790/0 conn(0x7fb3780a9d70 legacy=0x7fb3781deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 shutdown_connections 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 >> 172.21.15.117:0/2681624440 conn(0x7fb378066a00 msgr2=0x7fb37813b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 shutdown_connections 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 -- 172.21.15.117:0/2681624440 wait complete. 2024-03-29T11:28:11.123 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:11.133+0000 7fb37f3d6640 1 librados: shutdown 2024-03-29T11:28:11.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:17.944 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.100+0000 7fea993e9640 10 monclient: get_monmap_and_config 2024-03-29T11:28:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.100+0000 7fea993e9640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.090 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.100+0000 7fea993e9640 10 monclient: monmap: 2024-03-29T11:28:18.090 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:18.090 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:18.101567+0000 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:18.101567+0000 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:18.091 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.102+0000 7fea993e9640 1 Processor -- start 2024-03-29T11:28:18.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.102+0000 7fea993e9640 1 -- start start 2024-03-29T11:28:18.092 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.102+0000 7fea993e9640 10 monclient: init 2024-03-29T11:28:18.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:18.096 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-c con 0x7fea940b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-b con 0x7fea9414a700 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-a con 0x7fea940ac5a0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940aba70 con 0x7fea940ac5a0 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940aecb0 con 0x7fea940b0770 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940af150 con 0x7fea9414a700 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.103+0000 7fea993e9640 10 monclient(hunting): authenticate will time out at 1366.412148s 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.104+0000 7fea927fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fea9414a700 0x7fea940abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:36812/0 (socket says 172.21.15.117:36812) 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.105+0000 7fea927fc640 1 -- 172.21.15.117:0/630157569 learned_addr learned my addr 172.21.15.117:0/630157569 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396914689 0 0) 0x7fea940af150 con 0x7fea9414a700 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:28:18.097 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.106+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7c003520 con 0x7fea9414a700 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796751551 0 0) 0x7fea940aecb0 con 0x7fea940b0770 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:28:18.098 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.107+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea940af150 con 0x7fea940b0770 2024-03-29T11:28:18.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059356678 0 0) 0x7fea940af150 con 0x7fea940b0770 2024-03-29T11:28:18.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 >> v1:172.21.15.117:6790/0 conn(0x7fea9414a700 legacy=0x7fea940abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.099 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 >> v1:172.21.15.117:6789/0 conn(0x7fea940ac5a0 legacy=0x7fea94151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:18.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.101 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 1 monclient: found mon.noname-c 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940ac9d0 con 0x7fea940b0770 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.110+0000 7fea91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fea8c0019f0 con 0x7fea940b0770 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: dump: 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.102 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea8c001c10 con 0x7fea940b0770 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 1 -- 172.21.15.117:0/630157569 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fea8c001fa0 con 0x7fea940b0770 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea91ffb640 10 monclient: dump: 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.103 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.104 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.104 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:18.104 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:18.104 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:18.104 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea993e9640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:28:18.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea993e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:18.106 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.111+0000 7fea993e9640 4 set_mon_vals no callback set 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.112+0000 7fea993e9640 10 monclient: shutdown 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.112+0000 7fea993e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.112+0000 7fea993e9640 1 -- 172.21.15.117:0/630157569 >> v1:172.21.15.173:6789/0 conn(0x7fea940b0770 legacy=0x7fea94149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.112+0000 7fea993e9640 1 -- 172.21.15.117:0/630157569 shutdown_connections 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.112+0000 7fea993e9640 1 -- 172.21.15.117:0/630157569 >> 172.21.15.117:0/630157569 conn(0x7fea9406b660 msgr2=0x7fea9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.114+0000 7fea993e9640 1 -- 172.21.15.117:0/630157569 shutdown_connections 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.114+0000 7fea993e9640 1 -- 172.21.15.117:0/630157569 wait complete. 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 10 monclient: build_initial_monmap 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 10 monclient: monmap: 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:18.107 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 librados: starting msgr at 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 librados: starting objecter 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 Processor -- start 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 -- start start 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 librados: setting wanted keys 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 1 librados: calling monclient init 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.115+0000 7fea993e9640 10 monclient: init 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.116+0000 7fea993e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.116+0000 7fea993e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:18.108 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.116+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-c con 0x7fea940ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.116+0000 7fea92ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fea940ac5a0 0x7fea94126e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38826/0 (socket says 172.21.15.117:38826) 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.117+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-b con 0x7fea940b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea92ffd640 1 -- 172.21.15.117:0/911074124 learned_addr learned my addr 172.21.15.117:0/911074124 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 10 monclient(hunting): picked mon.noname-a con 0x7fea9414a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea94128260 con 0x7fea9414a700 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea941284d0 con 0x7fea940b0770 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea941db6d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea993e9640 10 monclient(hunting): authenticate will time out at 1366.428430s 2024-03-29T11:28:18.112 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea7b7fe640 10 client.?.objecter ms_handle_connect 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.119+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122534795 0 0) 0x7fea941db6d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80004eb0 con 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887226144 0 0) 0x7fea80004eb0 con 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea941db6d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fea8c002020 con 0x7fea940ac5a0 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:18.113 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 client.?.objecter ms_handle_connect 0x7fea940b0770 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.120+0000 7fea7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220335571 0 0) 0x7fea941284d0 con 0x7fea940b0770 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:18.114 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:18.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:28:18.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea80004eb0 con 0x7fea940b0770 2024-03-29T11:28:18.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208081642 0 0) 0x7fea941db6d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 >> v1:172.21.15.173:6789/0 conn(0x7fea940b0770 legacy=0x7fea941dbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.115 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.121+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 >> v1:172.21.15.117:6789/0 conn(0x7fea9414a700 legacy=0x7fea94145230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 1 monclient: found mon.c 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea941dc300 con 0x7fea940ac5a0 2024-03-29T11:28:18.117 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 10 monclient: _renew_subs 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea941239d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 10 monclient: _renew_subs 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.124+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea94145b20 con 0x7fea940ac5a0 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.125+0000 7fea993e9640 1 librados: init done 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.126+0000 7fea993e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.126+0000 7fea993e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.118 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.126+0000 7fea993e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.126+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea940ab560 con 0x7fea940ac5a0 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea8c003a70 con 0x7fea940ac5a0 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fea8c004080 con 0x7fea940ac5a0 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea7b7fe640 10 monclient: dump: 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:18.119 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:18.120 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:18.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.127+0000 7fea90ff9640 4 set_mon_vals no callback set 2024-03-29T11:28:18.120 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.128+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fea8c0042a0 con 0x7fea940ac5a0 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fea8c0850f0 con 0x7fea940ac5a0 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea7b7fe640 10 client.4646.objecter ms_dispatch 0x7fea94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea7b7fe640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea7b7fe640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea7b7fe640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.129+0000 7fea927fc640 10 monclient: get_auth_request con 0x7fea8006dcf0 auth_method 0 2024-03-29T11:28:18.121 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.130+0000 7fea7b7fe640 10 client.4646.objecter ms_handle_connect 0x7fea8006dcf0 2024-03-29T11:28:18.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.137+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea8c0509d0 con 0x7fea940ac5a0 2024-03-29T11:28:18.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.137+0000 7fea7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:18.127 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.137+0000 7fea7b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:18.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.460+0000 7fea993e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.460+0000 7fea993e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.460+0000 7fea993e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:28:18.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.460+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea941db530 con 0x7fea940ac5a0 2024-03-29T11:28:18.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.461+0000 7fea7b7fe640 1 -- 172.21.15.117:0/911074124 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fea941db530 con 0x7fea940ac5a0 2024-03-29T11:28:18.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.461+0000 7fea7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:18.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.461+0000 7fea7b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:18.452 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:18.452 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 10 librados: watch_flush enter 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 10 librados: watch_flush exit 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 >> v1:172.21.15.117:6816/4276038175 conn(0x7fea8006dcf0 legacy=0x7fea800701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 10 monclient: shutdown 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 >> v1:172.21.15.117:6790/0 conn(0x7fea940ac5a0 legacy=0x7fea94126e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 shutdown_connections 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 >> 172.21.15.117:0/911074124 conn(0x7fea9406b660 msgr2=0x7fea940b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 shutdown_connections 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.466+0000 7fea993e9640 1 -- 172.21.15.117:0/911074124 wait complete. 2024-03-29T11:28:18.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:18.467+0000 7fea993e9640 1 librados: shutdown 2024-03-29T11:28:18.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:25.284 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.451+0000 7fda0934d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.451+0000 7fda0934d640 10 monclient: build_initial_monmap 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.451+0000 7fda0934d640 10 monclient: monmap: 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:25.452833+0000 2024-03-29T11:28:25.444 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:25.452833+0000 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.454+0000 7fda0934d640 1 Processor -- start 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.454+0000 7fda0934d640 1 -- start start 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.454+0000 7fda0934d640 10 monclient: init 2024-03-29T11:28:25.445 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-a con 0x7fda040a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-c con 0x7fda0413ee90 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-b con 0x7fda040af900 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040afd10 con 0x7fda040a9ac0 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040af6f0 con 0x7fda0413ee90 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040aac80 con 0x7fda040af900 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:25.446 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.455+0000 7fda0934d640 10 monclient(hunting): authenticate will time out at 1373.764257s 2024-03-29T11:28:25.447 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.458+0000 7fda037fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fda040af900 0x7fda04149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:42383/0 (socket says 172.21.15.117:42383) 2024-03-29T11:28:25.448 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.458+0000 7fda037fe640 1 -- 172.21.15.117:0/3550272538 learned_addr learned my addr 172.21.15.117:0/3550272538 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463959203 0 0) 0x7fda040aac80 con 0x7fda040af900 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.459+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ec003520 con 0x7fda040af900 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388430269 0 0) 0x7fda040afd10 con 0x7fda040a9ac0 2024-03-29T11:28:25.449 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda040aac80 con 0x7fda040a9ac0 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418637765 0 0) 0x7fd9ec003520 con 0x7fda040af900 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 >> v1:172.21.15.173:6789/0 conn(0x7fda0413ee90 legacy=0x7fda0413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 >> v1:172.21.15.117:6789/0 conn(0x7fda040a9ac0 legacy=0x7fda040adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.450 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.460+0000 7fda01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040aaf60 con 0x7fda040af900 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9f8004c90 con 0x7fda040af900 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.461+0000 7fda01ffb640 10 monclient: dump: 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.451 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.463+0000 7fda0934d640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:28:25.452 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.463+0000 7fda0934d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.464+0000 7fda01ffb640 1 -- 172.21.15.117:0/3550272538 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f8004e70 con 0x7fda040af900 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.464+0000 7fda01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.464+0000 7fda0934d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.464+0000 7fda0934d640 4 set_mon_vals no callback set 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 10 monclient: shutdown 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 1 -- 172.21.15.117:0/3550272538 >> v1:172.21.15.117:6790/0 conn(0x7fda040af900 legacy=0x7fda04149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 1 -- 172.21.15.117:0/3550272538 shutdown_connections 2024-03-29T11:28:25.456 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 1 -- 172.21.15.117:0/3550272538 >> 172.21.15.117:0/3550272538 conn(0x7fda040b35e0 msgr2=0x7fda040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.465+0000 7fda0934d640 1 -- 172.21.15.117:0/3550272538 shutdown_connections 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.466+0000 7fda0934d640 1 -- 172.21.15.117:0/3550272538 wait complete. 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.466+0000 7fda0934d640 10 monclient: build_initial_monmap 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.466+0000 7fda0934d640 10 monclient: monmap: 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:25.457 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 librados: starting msgr at 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 librados: starting objecter 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 Processor -- start 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 -- start start 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 librados: setting wanted keys 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 1 librados: calling monclient init 2024-03-29T11:28:25.458 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.467+0000 7fda0934d640 10 monclient: init 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-c con 0x7fda040a9ac0 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-a con 0x7fda040af900 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient(hunting): picked mon.noname-b con 0x7fda0413ee90 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda04140340 con 0x7fda040af900 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda04140620 con 0x7fda0413ee90 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda04140d20 con 0x7fda040a9ac0 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:25.460 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.468+0000 7fda0934d640 10 monclient(hunting): authenticate will time out at 1373.777855s 2024-03-29T11:28:25.464 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.474+0000 7fda027fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fda040af900 0x7fda041412b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:36585/0 (socket says 172.21.15.117:36585) 2024-03-29T11:28:25.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.474+0000 7fda027fc640 1 -- 172.21.15.117:0/1334851177 learned_addr learned my addr 172.21.15.117:0/1334851177 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:25.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.474+0000 7fd9eb7fe640 10 client.?.objecter ms_handle_connect 0x7fda040af900 2024-03-29T11:28:25.466 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.474+0000 7fd9eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:25.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647298721 0 0) 0x7fda04140340 con 0x7fda040af900 2024-03-29T11:28:25.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.467 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:28:25.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.475+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f0004eb0 con 0x7fda040af900 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942860245 0 0) 0x7fd9f0004eb0 con 0x7fda040af900 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda04140340 con 0x7fda040af900 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9f40017d0 con 0x7fda040af900 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:25.470 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 client.?.objecter ms_handle_connect 0x7fda040a9ac0 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218706447 0 0) 0x7fda04140d20 con 0x7fda040a9ac0 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f0004eb0 con 0x7fda040a9ac0 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1063594110 0 0) 0x7fda04140340 con 0x7fda040af900 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 >> v1:172.21.15.117:6790/0 conn(0x7fda040a9ac0 legacy=0x7fda041dbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 >> v1:172.21.15.173:6789/0 conn(0x7fda0413ee90 legacy=0x7fda041419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 monclient: found mon.a 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0413f5c0 con 0x7fda040af900 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.476+0000 7fd9eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f4001930 con 0x7fda040af900 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd9f4004a00 con 0x7fda040af900 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fd9eb7fe640 10 monclient: dump: 2024-03-29T11:28:25.472 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.477+0000 7fda017fa640 4 set_mon_vals no callback set 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 10 monclient: _renew_subs 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda040b2130 con 0x7fda040af900 2024-03-29T11:28:25.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:25.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 10 monclient: _renew_subs 2024-03-29T11:28:25.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:25.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda040b3130 con 0x7fda040af900 2024-03-29T11:28:25.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.478+0000 7fda0934d640 1 librados: init done 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.480+0000 7fda0934d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.480+0000 7fda0934d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.480+0000 7fda0934d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.480+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04142700 con 0x7fda040af900 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.483+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd9f401a9d0 con 0x7fda040af900 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fd9f4084980 con 0x7fda040af900 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 10 client.4533.objecter ms_dispatch 0x7fda04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:25.475 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:25.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:28:25.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fda02ffd640 10 monclient: get_auth_request con 0x7fd9f006dc60 auth_method 0 2024-03-29T11:28:25.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.484+0000 7fd9eb7fe640 10 client.4533.objecter ms_handle_connect 0x7fd9f006dc60 2024-03-29T11:28:25.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.491+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9f4050090 con 0x7fda040af900 2024-03-29T11:28:25.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.491+0000 7fd9eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:25.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.491+0000 7fd9eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:25.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.826+0000 7fda0934d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:25.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.826+0000 7fda0934d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:25.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.826+0000 7fda0934d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:25.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.826+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda04075f10 con 0x7fda040af900 2024-03-29T11:28:25.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.832+0000 7fd9eb7fe640 1 -- 172.21.15.117:0/1334851177 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fd9f4052190 con 0x7fda040af900 2024-03-29T11:28:25.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.832+0000 7fd9eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:25.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.832+0000 7fd9eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:25.822 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:25.823 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:25.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.836+0000 7fda0934d640 10 librados: watch_flush enter 2024-03-29T11:28:25.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 10 librados: watch_flush exit 2024-03-29T11:28:25.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:28:25.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd9f006dc60 legacy=0x7fd9f0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 10 monclient: shutdown 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 >> v1:172.21.15.117:6789/0 conn(0x7fda040af900 legacy=0x7fda041412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 shutdown_connections 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 >> 172.21.15.117:0/1334851177 conn(0x7fda040b35e0 msgr2=0x7fda0413b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 shutdown_connections 2024-03-29T11:28:25.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 -- 172.21.15.117:0/1334851177 wait complete. 2024-03-29T11:28:25.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:25.839+0000 7fda0934d640 1 librados: shutdown 2024-03-29T11:28:25.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:32.654 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:32.800 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.809+0000 7f7a479ef640 10 monclient: get_monmap_and_config 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.809+0000 7f7a479ef640 10 monclient: build_initial_monmap 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.809+0000 7f7a479ef640 10 monclient: monmap: 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:32.810748+0000 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:32.810748+0000 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:32.801 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.810+0000 7f7a479ef640 1 Processor -- start 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 1 -- start start 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient: init 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:32.802 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-a con 0x7f7a400a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4013ee90 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.811+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-c con 0x7f7a400af900 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a479ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400afd10 con 0x7f7a400a9ac0 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a479ef640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400af6f0 con 0x7f7a400af900 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a479ef640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400aac80 con 0x7f7a4013ee90 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a479ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a479ef640 10 monclient(hunting): authenticate will time out at 1381.121041s 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a45f8c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7a400af900 0x7f7a40149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:39856/0 (socket says 172.21.15.117:39856) 2024-03-29T11:28:32.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.812+0000 7f7a45f8c640 1 -- 172.21.15.117:0/850164797 learned_addr learned my addr 172.21.15.117:0/850164797 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:32.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716103051 0 0) 0x7f7a400af6f0 con 0x7f7a400af900 2024-03-29T11:28:32.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:32.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:32.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:32.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.814+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c003520 con 0x7f7a400af900 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.815+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2293944893 0 0) 0x7f7a2c003520 con 0x7f7a400af900 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.815+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 >> v1:172.21.15.117:6790/0 conn(0x7f7a4013ee90 legacy=0x7f7a4013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.815+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 >> v1:172.21.15.117:6789/0 conn(0x7f7a400a9ac0 legacy=0x7f7a400adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 1 monclient: found mon.noname-c 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400aaf60 con 0x7f7a400af900 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a479ef640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a479ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:32.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7a30001690 con 0x7f7a400af900 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.816+0000 7f7a37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.817+0000 7f7a37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.817+0000 7f7a37fff640 10 monclient: dump: 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:32.809 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.817+0000 7f7a479ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:32.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.818+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a30001920 con 0x7f7a400af900 2024-03-29T11:28:32.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.818+0000 7f7a37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:32.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a37fff640 1 -- 172.21.15.117:0/850164797 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7a30001cb0 con 0x7f7a400af900 2024-03-29T11:28:32.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:32.811 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a37fff640 10 monclient: dump: 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a479ef640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.819+0000 7f7a479ef640 4 set_mon_vals no callback set 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 10 monclient: shutdown 2024-03-29T11:28:32.812 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 1 -- 172.21.15.117:0/850164797 >> v1:172.21.15.173:6789/0 conn(0x7f7a400af900 legacy=0x7f7a40149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 1 -- 172.21.15.117:0/850164797 shutdown_connections 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 1 -- 172.21.15.117:0/850164797 >> 172.21.15.117:0/850164797 conn(0x7f7a400b35e0 msgr2=0x7f7a400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 1 -- 172.21.15.117:0/850164797 shutdown_connections 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.820+0000 7f7a479ef640 1 -- 172.21.15.117:0/850164797 wait complete. 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.821+0000 7f7a479ef640 10 monclient: build_initial_monmap 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.821+0000 7f7a479ef640 10 monclient: monmap: 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:32.813 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.821+0000 7f7a479ef640 1 librados: starting msgr at 2024-03-29T11:28:32.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.821+0000 7f7a479ef640 1 librados: starting objecter 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 1 Processor -- start 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 1 -- start start 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 1 librados: setting wanted keys 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 1 librados: calling monclient init 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 10 monclient: init 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-a con 0x7f7a400a9ac0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:32.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.822+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-c con 0x7f7a400af900 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.826+0000 7f7a44f8a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f7a400af900 0x7f7a401f8250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38175/0 (socket says 172.21.15.117:38175) 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.826+0000 7f7a45f8c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7a4013ee90 0x7f7a40137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:42504/0 (socket says 172.21.15.117:42504) 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.826+0000 7f7a45f8c640 1 -- 172.21.15.117:0/1669852176 learned_addr learned my addr 172.21.15.117:0/1669852176 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.826+0000 7f7a479ef640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4013ee90 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.826+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4006f020 con 0x7f7a400a9ac0 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a40072170 con 0x7f7a4013ee90 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4006f4b0 con 0x7f7a400af900 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a479ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a479ef640 10 monclient(hunting): authenticate will time out at 1381.136275s 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 client.?.objecter ms_handle_connect 0x7f7a4013ee90 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:32.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 client.?.objecter ms_handle_connect 0x7f7a400af900 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886998975 0 0) 0x7f7a40072170 con 0x7f7a4013ee90 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.827+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a24004eb0 con 0x7f7a4013ee90 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 client.?.objecter ms_handle_connect 0x7f7a400a9ac0 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266428691 0 0) 0x7f7a4006f4b0 con 0x7f7a400af900 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:32.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a40072170 con 0x7f7a400af900 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666312871 0 0) 0x7f7a4006f020 con 0x7f7a400a9ac0 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4006f4b0 con 0x7f7a400a9ac0 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1008863612 0 0) 0x7f7a24004eb0 con 0x7f7a4013ee90 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4006f020 con 0x7f7a4013ee90 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2700997912 0 0) 0x7f7a40072170 con 0x7f7a400af900 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.828+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a24004eb0 con 0x7f7a400af900 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7a30001550 con 0x7f7a4013ee90 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:32.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7a38001aa0 con 0x7f7a400af900 2024-03-29T11:28:32.826 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 523954007 0 0) 0x7f7a4006f020 con 0x7f7a4013ee90 2024-03-29T11:28:32.827 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 >> v1:172.21.15.117:6790/0 conn(0x7f7a400af900 legacy=0x7f7a401f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 >> v1:172.21.15.117:6789/0 conn(0x7f7a400a9ac0 legacy=0x7f7a4006e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 monclient: found mon.b 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4006f640 con 0x7f7a4013ee90 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.829+0000 7f7a35ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a35ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 10 monclient: _renew_subs 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a401f8960 con 0x7f7a4013ee90 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:32.828 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 10 monclient: _renew_subs 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40141560 con 0x7f7a4013ee90 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.830+0000 7f7a479ef640 1 librados: init done 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a30004bf0 con 0x7f7a4013ee90 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7a30004f00 con 0x7f7a4013ee90 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.831+0000 7f7a35ffb640 10 monclient: dump: 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:32.829 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.832+0000 7f7a377fe640 4 set_mon_vals no callback set 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.832+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7a3001aed0 con 0x7f7a4013ee90 2024-03-29T11:28:32.830 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.833+0000 7f7a4578b640 10 monclient: get_auth_request con 0x7f7a2406df00 auth_method 0 2024-03-29T11:28:32.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 10 client.4543.objecter ms_handle_connect 0x7f7a2406df00 2024-03-29T11:28:32.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f7a30084a60 con 0x7f7a4013ee90 2024-03-29T11:28:32.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 10 client.4543.objecter ms_dispatch 0x7f7a40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:32.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:32.831 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:32.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a35ffb640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:28:32.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a479ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:32.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a479ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:32.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a479ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:32.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.834+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20005080 con 0x7f7a4013ee90 2024-03-29T11:28:32.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.843+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a30050340 con 0x7f7a4013ee90 2024-03-29T11:28:32.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.843+0000 7f7a35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:32.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:32.843+0000 7f7a35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:33.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.160+0000 7f7a479ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.160+0000 7f7a479ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.160+0000 7f7a479ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:28:33.151 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.160+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a20005330 con 0x7f7a4013ee90 2024-03-29T11:28:33.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.165+0000 7f7a35ffb640 1 -- 172.21.15.117:0/1669852176 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f7a30052270 con 0x7f7a4013ee90 2024-03-29T11:28:33.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.165+0000 7f7a35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:33.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.165+0000 7f7a35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:33.156 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:33.156 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:33.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.168+0000 7f7a479ef640 10 librados: watch_flush enter 2024-03-29T11:28:33.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.169+0000 7f7a479ef640 10 librados: watch_flush exit 2024-03-29T11:28:33.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.169+0000 7f7a479ef640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:28:33.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.169+0000 7f7a479ef640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:28:33.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.170+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 >> v1:172.21.15.117:6816/4276038175 conn(0x7f7a2406df00 legacy=0x7f7a240703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.170+0000 7f7a479ef640 10 monclient: shutdown 2024-03-29T11:28:33.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.170+0000 7f7a479ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:33.160 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.170+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 >> v1:172.21.15.173:6789/0 conn(0x7f7a4013ee90 legacy=0x7f7a40137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:33.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.172+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 shutdown_connections 2024-03-29T11:28:33.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.172+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 >> 172.21.15.117:0/1669852176 conn(0x7f7a400b35e0 msgr2=0x7f7a4013b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:33.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.172+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 shutdown_connections 2024-03-29T11:28:33.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.172+0000 7f7a479ef640 1 -- 172.21.15.117:0/1669852176 wait complete. 2024-03-29T11:28:33.162 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:33.172+0000 7f7a479ef640 1 librados: shutdown 2024-03-29T11:28:33.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:39.984 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:40.132 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.142+0000 7efcb4ea8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:40.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.142+0000 7efcb4ea8640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.133 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.142+0000 7efcb4ea8640 10 monclient: monmap: 2024-03-29T11:28:40.133 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:40.133 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:40.133 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:40.143774+0000 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:40.143774+0000 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:40.134 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:40.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.144+0000 7efcb4ea8640 1 Processor -- start 2024-03-29T11:28:40.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.144+0000 7efcb4ea8640 1 -- start start 2024-03-29T11:28:40.136 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.144+0000 7efcb4ea8640 10 monclient: init 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7efcb00b1f60 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7efcb0071eb0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7efcb0143d10 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb006ef80 con 0x7efcb0071eb0 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb006f260 con 0x7efcb0143d10 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00b40c0 con 0x7efcb00b1f60 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.145+0000 7efcb4ea8640 10 monclient(hunting): authenticate will time out at 1388.454929s 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcaed9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7efcb0143d10 0x7efcb0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:40038/0 (socket says 172.21.15.117:40038) 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcaed9d640 1 -- 172.21.15.117:0/86193714 learned_addr learned my addr 172.21.15.117:0/86193714 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532083246 0 0) 0x7efcb006f260 con 0x7efcb0143d10 2024-03-29T11:28:40.137 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.146+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc98003520 con 0x7efcb0143d10 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.148+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448986229 0 0) 0x7efc98003520 con 0x7efcb0143d10 2024-03-29T11:28:40.138 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.148+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 >> v1:172.21.15.117:6790/0 conn(0x7efcb00b1f60 legacy=0x7efcb0143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:40.139 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.149+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 >> v1:172.21.15.117:6789/0 conn(0x7efcb0071eb0 legacy=0x7efcb0146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 1 monclient: found mon.noname-c 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0155d70 con 0x7efcb0143d10 2024-03-29T11:28:40.141 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7efca4001670 con 0x7efcb0143d10 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: dump: 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:40.142 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcb4ea8640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcb4ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 1 -- 172.21.15.117:0/86193714 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efca40018d0 con 0x7efcb0143d10 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcb4ea8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:40.143 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.151+0000 7efcb4ea8640 4 set_mon_vals no callback set 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 10 monclient: shutdown 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 1 -- 172.21.15.117:0/86193714 >> v1:172.21.15.173:6789/0 conn(0x7efcb0143d10 legacy=0x7efcb0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 1 -- 172.21.15.117:0/86193714 shutdown_connections 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 1 -- 172.21.15.117:0/86193714 >> 172.21.15.117:0/86193714 conn(0x7efcb0066a00 msgr2=0x7efcb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 1 -- 172.21.15.117:0/86193714 shutdown_connections 2024-03-29T11:28:40.144 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.152+0000 7efcb4ea8640 1 -- 172.21.15.117:0/86193714 wait complete. 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 10 monclient: build_initial_monmap 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 10 monclient: monmap: 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 librados: starting msgr at 2024-03-29T11:28:40.145 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 librados: starting objecter 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 Processor -- start 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 -- start start 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 librados: setting wanted keys 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 1 librados: calling monclient init 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.154+0000 7efcb4ea8640 10 monclient: init 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:40.146 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7efcb00b1f60 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7efcb0143d10 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7efcb0149e00 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb0149970 con 0x7efcb0149e00 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb014af00 con 0x7efcb00b1f60 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb014b1e0 con 0x7efcb0143d10 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:40.147 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.155+0000 7efcb4ea8640 10 monclient(hunting): authenticate will time out at 1388.464461s 2024-03-29T11:28:40.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efcaed9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7efcb0149e00 0x7efcb01e6270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:36801/0 (socket says 172.21.15.117:36801) 2024-03-29T11:28:40.152 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efcaed9d640 1 -- 172.21.15.117:0/462326090 learned_addr learned my addr 172.21.15.117:0/462326090 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efcb0149e00 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efcb00b1f60 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.158+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 client.?.objecter ms_handle_connect 0x7efcb0143d10 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514459913 0 0) 0x7efcb0149970 con 0x7efcb0149e00 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc90004eb0 con 0x7efcb0149e00 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705150009 0 0) 0x7efcb014b1e0 con 0x7efcb0143d10 2024-03-29T11:28:40.153 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb0149970 con 0x7efcb0143d10 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671163509 0 0) 0x7efc90004eb0 con 0x7efcb0149e00 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcb014b1e0 con 0x7efcb0149e00 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7efca4001630 con 0x7efcb0149e00 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): dump: 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:40.154 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2046385087 0 0) 0x7efcb014b1e0 con 0x7efcb0149e00 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.159+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 >> v1:172.21.15.117:6790/0 conn(0x7efcb0143d10 legacy=0x7efcb01475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 >> v1:172.21.15.173:6789/0 conn(0x7efcb00b1f60 legacy=0x7efcb01490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 1 monclient: found mon.a 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:40.155 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0147ce0 con 0x7efcb0149e00 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efc9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 10 monclient: _renew_subs 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb0070890 con 0x7efcb0149e00 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 10 monclient: _renew_subs 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb01e7990 con 0x7efcb0149e00 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.160+0000 7efcb4ea8640 1 librados: init done 2024-03-29T11:28:40.156 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.161+0000 7efcb4ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.161+0000 7efcb4ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.161+0000 7efcb4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.161+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb0075f10 con 0x7efcb0149e00 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efca4001b80 con 0x7efcb0149e00 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7efca4005090 con 0x7efcb0149e00 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.162+0000 7efc9effd640 10 monclient: dump: 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:40.157 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.163+0000 7efc9ffff640 4 set_mon_vals no callback set 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7efca401b060 con 0x7efcb0149e00 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efcae59c640 10 monclient: get_auth_request con 0x7efc9006df20 auth_method 0 2024-03-29T11:28:40.158 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7efca4054830 con 0x7efcb0149e00 2024-03-29T11:28:40.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 10 client.4539.objecter ms_dispatch 0x7efcb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:40.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:40.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:40.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.166+0000 7efc9effd640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:28:40.159 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.167+0000 7efc9effd640 10 client.4539.objecter ms_handle_connect 0x7efc9006df20 2024-03-29T11:28:40.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.171+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efca4050ad0 con 0x7efcb0149e00 2024-03-29T11:28:40.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.171+0000 7efc9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:40.161 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.171+0000 7efc9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:40.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.490+0000 7efcb4ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.490+0000 7efcb4ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.490+0000 7efcb4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.490+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcb00b6330 con 0x7efcb0149e00 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.491+0000 7efc9effd640 1 -- 172.21.15.117:0/462326090 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7efca4050c70 con 0x7efcb0149e00 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.491+0000 7efc9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.491+0000 7efc9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:40.481 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:40.482 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:40.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 10 librados: watch_flush enter 2024-03-29T11:28:40.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 10 librados: watch_flush exit 2024-03-29T11:28:40.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 >> v1:172.21.15.117:6816/4276038175 conn(0x7efc9006df20 legacy=0x7efc900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 10 monclient: shutdown 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 >> v1:172.21.15.117:6789/0 conn(0x7efcb0149e00 legacy=0x7efcb01e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 shutdown_connections 2024-03-29T11:28:40.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.494+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 >> 172.21.15.117:0/462326090 conn(0x7efcb0066a00 msgr2=0x7efcb01468c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:40.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.495+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 shutdown_connections 2024-03-29T11:28:40.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.495+0000 7efcb4ea8640 1 -- 172.21.15.117:0/462326090 wait complete. 2024-03-29T11:28:40.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:40.495+0000 7efcb4ea8640 1 librados: shutdown 2024-03-29T11:28:40.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:47.310 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:47.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.465+0000 7fbb6ecd0640 10 monclient: get_monmap_and_config 2024-03-29T11:28:47.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.465+0000 7fbb6ecd0640 10 monclient: build_initial_monmap 2024-03-29T11:28:47.457 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.465+0000 7fbb6ecd0640 10 monclient: monmap: 2024-03-29T11:28:47.457 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:47.457 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:47.466688+0000 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:47.466688+0000 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:47.458 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.466+0000 7fbb6ecd0640 1 Processor -- start 2024-03-29T11:28:47.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.466+0000 7fbb6ecd0640 1 -- start start 2024-03-29T11:28:47.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.466+0000 7fbb6ecd0640 10 monclient: init 2024-03-29T11:28:47.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:47.469 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-a con 0x7fbb681422b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbb68142690 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbb680b17d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb68142c80 con 0x7fbb681422b0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb680aae40 con 0x7fbb68142690 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6813fbd0 con 0x7fbb680b17d0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ecd0640 10 monclient(hunting): authenticate will time out at 1395.777281s 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6d26d640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fbb680b17d0 0x7fbb680b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:49898/0 (socket says 172.21.15.117:49898) 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6ca6c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbb681422b0 0x7fbb6813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:42158/0 (socket says 172.21.15.117:42158) 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb6d26d640 1 -- 172.21.15.117:0/2243208833 learned_addr learned my addr 172.21.15.117:0/2243208833 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103081342 0 0) 0x7fbb6813fbd0 con 0x7fbb680b17d0 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.470 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.468+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb680b17d0 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100548810 0 0) 0x7fbb68142c80 con 0x7fbb681422b0 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6813fbd0 con 0x7fbb681422b0 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937884369 0 0) 0x7fbb680aae40 con 0x7fbb68142690 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.471 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb68142c80 con 0x7fbb68142690 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472211818 0 0) 0x7fbb50003520 con 0x7fbb680b17d0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 >> v1:172.21.15.173:6789/0 conn(0x7fbb68142690 legacy=0x7fbb680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 >> v1:172.21.15.117:6789/0 conn(0x7fbb681422b0 legacy=0x7fbb6813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.469+0000 7fbb5f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68151940 con 0x7fbb680b17d0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb60001690 con 0x7fbb680b17d0 2024-03-29T11:28:47.472 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: dump: 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb6ecd0640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb6ecd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb600017f0 con 0x7fbb680b17d0 2024-03-29T11:28:47.473 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 1 -- 172.21.15.117:0/2243208833 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb60001b80 con 0x7fbb680b17d0 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb6ecd0640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb6ecd0640 4 set_mon_vals no callback set 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.470+0000 7fbb5f7fe640 10 monclient: dump: 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.474 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.475 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.471+0000 7fbb6ecd0640 10 monclient: shutdown 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.471+0000 7fbb6ecd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.471+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/2243208833 >> v1:172.21.15.117:6790/0 conn(0x7fbb680b17d0 legacy=0x7fbb680b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.471+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/2243208833 shutdown_connections 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.471+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/2243208833 >> 172.21.15.117:0/2243208833 conn(0x7fbb68066a00 msgr2=0x7fbb68066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.472+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/2243208833 shutdown_connections 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.472+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/2243208833 wait complete. 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.472+0000 7fbb6ecd0640 10 monclient: build_initial_monmap 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.472+0000 7fbb6ecd0640 10 monclient: monmap: 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:47.476 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 librados: starting msgr at 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 librados: starting objecter 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 Processor -- start 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 -- start start 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 librados: setting wanted keys 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 1 librados: calling monclient init 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.473+0000 7fbb6ecd0640 10 monclient: init 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.474+0000 7fbb6ecd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.474+0000 7fbb6ecd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:47.477 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.474+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-a con 0x7fbb680b17d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbb681422b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5ffff640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fbb681422b0 0x7fbb680709f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:51018/0 (socket says 172.21.15.117:51018) 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5ffff640 1 -- 172.21.15.117:0/3218001760 learned_addr learned my addr 172.21.15.117:0/3218001760 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbb68142690 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6806fa60 con 0x7fbb680b17d0 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6814e5d0 con 0x7fbb68142690 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6806fcd0 con 0x7fbb681422b0 2024-03-29T11:28:47.478 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb6ecd0640 10 monclient(hunting): authenticate will time out at 1395.784367s 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 client.?.objecter ms_handle_connect 0x7fbb680b17d0 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 client.?.objecter ms_handle_connect 0x7fbb681422b0 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299707957 0 0) 0x7fbb6806fa60 con 0x7fbb680b17d0 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.475+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb680b17d0 2024-03-29T11:28:47.479 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510561741 0 0) 0x7fbb6806fcd0 con 0x7fbb681422b0 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6806fa60 con 0x7fbb681422b0 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 client.?.objecter ms_handle_connect 0x7fbb68142690 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939862873 0 0) 0x7fbb6814e5d0 con 0x7fbb68142690 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:47.480 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6806fcd0 con 0x7fbb68142690 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527192157 0 0) 0x7fbb54004eb0 con 0x7fbb680b17d0 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb6814e5d0 con 0x7fbb680b17d0 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb64001c00 con 0x7fbb680b17d0 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.481 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694774842 0 0) 0x7fbb6806fa60 con 0x7fbb681422b0 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb681422b0 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb58001980 con 0x7fbb681422b0 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.482 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220419138 0 0) 0x7fbb6806fcd0 con 0x7fbb68142690 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb6806fa60 con 0x7fbb68142690 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb60001100 con 0x7fbb68142690 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:47.483 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.476+0000 7fbb5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3589700984 0 0) 0x7fbb6814e5d0 con 0x7fbb680b17d0 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 >> v1:172.21.15.117:6790/0 conn(0x7fbb681422b0 legacy=0x7fbb680709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 >> v1:172.21.15.173:6789/0 conn(0x7fbb68142690 legacy=0x7fbb68071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:47.484 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 1 monclient: found mon.a 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6806ffb0 con 0x7fbb680b17d0 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:28:47.485 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 10 monclient: _renew_subs 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb681dfa60 con 0x7fbb680b17d0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 10 monclient: _renew_subs 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb681deec0 con 0x7fbb680b17d0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.477+0000 7fbb6ecd0640 1 librados: init done 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb64004b60 con 0x7fbb680b17d0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5effd640 4 set_mon_vals no callback set 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fbb64004e70 con 0x7fbb680b17d0 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:47.486 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb5d7fa640 10 monclient: dump: 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:47.487 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.478+0000 7fbb6ecd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.479+0000 7fbb6ecd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.479+0000 7fbb6ecd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:47.488 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.479+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48005080 con 0x7fbb680b17d0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.480+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fbb6401ae40 con 0x7fbb680b17d0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fbb64084cf0 con 0x7fbb680b17d0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5d7fa640 10 client.4545.objecter ms_dispatch 0x7fbb68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5d7fa640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5d7fa640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5ffff640 10 monclient: get_auth_request con 0x7fbb5406e0d0 auth_method 0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.481+0000 7fbb5d7fa640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.482+0000 7fbb5d7fa640 10 client.4545.objecter ms_handle_connect 0x7fbb5406e0d0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.489+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb640505d0 con 0x7fbb680b17d0 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.489+0000 7fbb5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:47.489 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.489+0000 7fbb5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb6ecd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb6ecd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb6ecd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb48005330 con 0x7fbb680b17d0 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb5d7fa640 1 -- 172.21.15.117:0/3218001760 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fbb64052500 con 0x7fbb680b17d0 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:47.829 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.838+0000 7fbb5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:47.830 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:47.830 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:47.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 10 librados: watch_flush enter 2024-03-29T11:28:47.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 10 librados: watch_flush exit 2024-03-29T11:28:47.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:28:47.832 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 >> v1:172.21.15.117:6816/4276038175 conn(0x7fbb5406e0d0 legacy=0x7fbb54070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 10 monclient: shutdown 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 >> v1:172.21.15.117:6789/0 conn(0x7fbb680b17d0 legacy=0x7fbb6813bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 shutdown_connections 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 >> 172.21.15.117:0/3218001760 conn(0x7fbb68066a00 msgr2=0x7fbb6813d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 shutdown_connections 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.841+0000 7fbb6ecd0640 1 -- 172.21.15.117:0/3218001760 wait complete. 2024-03-29T11:28:47.833 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:47.842+0000 7fbb6ecd0640 1 librados: shutdown 2024-03-29T11:28:47.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:54.653 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.810+0000 7f04f539a640 10 monclient: get_monmap_and_config 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.810+0000 7f04f539a640 10 monclient: build_initial_monmap 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.810+0000 7f04f539a640 10 monclient: monmap: 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:28:54.811612+0000 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:28:54.811612+0000 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.801 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:54.802 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:28:54.802 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:28:54.802 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.811+0000 7f04f539a640 1 Processor -- start 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.811+0000 7f04f539a640 1 -- start start 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.811+0000 7f04f539a640 10 monclient: init 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-a con 0x7f04f01457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-b con 0x7f04f013f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-c con 0x7f04f00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:54.803 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00aaef0 con 0x7f04f01457f0 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f0145bd0 con 0x7f04f00a9d70 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f0145eb0 con 0x7f04f013f650 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04f539a640 10 monclient(hunting): authenticate will time out at 1403.121394s 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04ee7fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f04f013f650 0x7f04f00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:55590/0 (socket says 172.21.15.117:55590) 2024-03-29T11:28:54.804 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.812+0000 7f04ee7fc640 1 -- 172.21.15.117:0/1493854340 learned_addr learned my addr 172.21.15.117:0/1493854340 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934939231 0 0) 0x7f04f0145eb0 con 0x7f04f013f650 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d8003520 con 0x7f04f013f650 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159963850 0 0) 0x7f04f00aaef0 con 0x7f04f01457f0 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.806 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f0145eb0 con 0x7f04f01457f0 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778441613 0 0) 0x7f04f0145bd0 con 0x7f04f00a9d70 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.813+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f00aaef0 con 0x7f04f00a9d70 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4076187263 0 0) 0x7f04f00aaef0 con 0x7f04f00a9d70 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 >> v1:172.21.15.117:6790/0 conn(0x7f04f013f650 legacy=0x7f04f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 >> v1:172.21.15.117:6789/0 conn(0x7f04f01457f0 legacy=0x7f04f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 monclient: found mon.noname-c 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:28:54.807 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0146340 con 0x7f04f00a9d70 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e0001630 con 0x7f04f00a9d70 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: dump: 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.808 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04f539a640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04f539a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e0001850 con 0x7f04f00a9d70 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 1 -- 172.21.15.117:0/1493854340 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e0001be0 con 0x7f04f00a9d70 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04edffb640 10 monclient: dump: 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.809 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04f539a640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.814+0000 7f04f539a640 4 set_mon_vals no callback set 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 10 monclient: shutdown 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 1 -- 172.21.15.117:0/1493854340 >> v1:172.21.15.173:6789/0 conn(0x7f04f00a9d70 legacy=0x7f04f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 1 -- 172.21.15.117:0/1493854340 shutdown_connections 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 1 -- 172.21.15.117:0/1493854340 >> 172.21.15.117:0/1493854340 conn(0x7f04f0066a00 msgr2=0x7f04f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 1 -- 172.21.15.117:0/1493854340 shutdown_connections 2024-03-29T11:28:54.810 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.815+0000 7f04f539a640 1 -- 172.21.15.117:0/1493854340 wait complete. 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.816+0000 7f04f539a640 10 monclient: build_initial_monmap 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.816+0000 7f04f539a640 10 monclient: monmap: 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:54.813 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.816+0000 7f04f539a640 1 librados: starting msgr at 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.816+0000 7f04f539a640 1 librados: starting objecter 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 1 Processor -- start 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 1 -- start start 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 1 librados: setting wanted keys 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 1 librados: calling monclient init 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 10 monclient: init 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:54.814 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-b con 0x7f04f00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.817+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-a con 0x7f04f013f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 10 monclient(hunting): picked mon.noname-c con 0x7f04f01457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f0071d40 con 0x7f04f013f650 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f0141b30 con 0x7f04f00a9d70 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f0141e10 con 0x7f04f01457f0 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04f539a640 10 monclient(hunting): authenticate will time out at 1403.127038s 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04eeffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f04f00a9d70 0x7f04f01425f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:37212/0 (socket says 172.21.15.117:37212) 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04eeffd640 1 -- 172.21.15.117:0/665020291 learned_addr learned my addr 172.21.15.117:0/665020291 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 client.?.objecter ms_handle_connect 0x7f04f00a9d70 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 client.?.objecter ms_handle_connect 0x7f04f013f650 2024-03-29T11:28:54.815 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952699171 0 0) 0x7f04f0141b30 con 0x7f04f00a9d70 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.818+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc004eb0 con 0x7f04f00a9d70 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 client.?.objecter ms_handle_connect 0x7f04f01457f0 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408120387 0 0) 0x7f04f0071d40 con 0x7f04f013f650 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f0141b30 con 0x7f04f013f650 2024-03-29T11:28:54.816 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483868014 0 0) 0x7f04f0141e10 con 0x7f04f01457f0 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.819+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f0071d40 con 0x7f04f01457f0 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951952163 0 0) 0x7f04f0071d40 con 0x7f04f01457f0 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f0141e10 con 0x7f04f01457f0 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578260224 0 0) 0x7f04dc004eb0 con 0x7f04f00a9d70 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f0071d40 con 0x7f04f00a9d70 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183631915 0 0) 0x7f04f0141b30 con 0x7f04f013f650 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04dc004eb0 con 0x7f04f013f650 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e0001570 con 0x7f04f01457f0 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:54.817 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): dump: 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e8001c90 con 0x7f04f00a9d70 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:54.818 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): dump: 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e40018d0 con 0x7f04f013f650 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.819 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient(hunting): dump: 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2566090539 0 0) 0x7f04dc004eb0 con 0x7f04f013f650 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 >> v1:172.21.15.117:6790/0 conn(0x7f04f01457f0 legacy=0x7f04f01e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 >> v1:172.21.15.173:6789/0 conn(0x7f04f00a9d70 legacy=0x7f04f01425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.820 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 monclient: found mon.a 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0142d00 con 0x7f04f013f650 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 10 monclient: _renew_subs 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f013b7c0 con 0x7f04f013f650 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 10 monclient: _renew_subs 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:54.821 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f01399a0 con 0x7f04f013f650 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04f539a640 1 librados: init done 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e40047e0 con 0x7f04f013f650 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f04e4004af0 con 0x7f04f013f650 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.820+0000 7f04d77fe640 10 monclient: dump: 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:28:54.822 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:28:54.823 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:28:54.823 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:28:54.823 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.821+0000 7f04ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.823+0000 7f04f539a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.823+0000 7f04f539a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.823+0000 7f04f539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.823+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d0005080 con 0x7f04f013f650 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.823+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f04e401aac0 con 0x7f04f013f650 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7f04e4084940 con 0x7f04f013f650 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04d77fe640 10 client.4551.objecter ms_dispatch 0x7f04f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04d77fe640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04d77fe640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:54.824 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04d77fe640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:28:54.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.824+0000 7f04eeffd640 10 monclient: get_auth_request con 0x7f04dc06e210 auth_method 0 2024-03-29T11:28:54.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.825+0000 7f04d77fe640 10 client.4551.objecter ms_handle_connect 0x7f04dc06e210 2024-03-29T11:28:54.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.833+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04e4050220 con 0x7f04f013f650 2024-03-29T11:28:54.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.833+0000 7f04d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:54.825 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:54.833+0000 7f04d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:55.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.228+0000 7f04f539a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.228+0000 7f04f539a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.228+0000 7f04f539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:28:55.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.228+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04d0005330 con 0x7f04f013f650 2024-03-29T11:28:55.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.229+0000 7f04d77fe640 1 -- 172.21.15.117:0/665020291 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7f04e4052150 con 0x7f04f013f650 2024-03-29T11:28:55.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.229+0000 7f04d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:55.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.229+0000 7f04d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:55.219 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:28:55.219 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 10 librados: watch_flush enter 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 10 librados: watch_flush exit 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 >> v1:172.21.15.117:6816/4276038175 conn(0x7f04dc06e210 legacy=0x7f04dc0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 10 monclient: shutdown 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:55.223 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 >> v1:172.21.15.117:6789/0 conn(0x7f04f013f650 legacy=0x7f04f01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:55.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 shutdown_connections 2024-03-29T11:28:55.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.233+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 >> 172.21.15.117:0/665020291 conn(0x7f04f0066a00 msgr2=0x7f04f00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:55.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.234+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 shutdown_connections 2024-03-29T11:28:55.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.234+0000 7f04f539a640 1 -- 172.21.15.117:0/665020291 wait complete. 2024-03-29T11:28:55.224 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:28:55.234+0000 7f04f539a640 1 librados: shutdown 2024-03-29T11:28:55.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:02.047 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.212+0000 7fdcf52a8640 10 monclient: get_monmap_and_config 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.212+0000 7fdcf52a8640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.212+0000 7fdcf52a8640 10 monclient: monmap: 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:02.206 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:02.213186+0000 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:02.213186+0000 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 1 Processor -- start 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 1 -- start start 2024-03-29T11:29:02.207 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 10 monclient: init 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.213+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf006d9d0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf013e820 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf00af1d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00aed10 con 0x7fdcf006d9d0 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00afd50 con 0x7fdcf013e820 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00ab490 con 0x7fdcf00af1d0 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.215 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.214+0000 7fdcf52a8640 10 monclient(hunting): authenticate will time out at 1410.523074s 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.216+0000 7fdceeffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdcf006d9d0 0x7fdcf00b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:37857/0 (socket says 172.21.15.117:37857) 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.216+0000 7fdceeffd640 1 -- 172.21.15.117:0/1820459235 learned_addr learned my addr 172.21.15.117:0/1820459235 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930496541 0 0) 0x7fdcf00aed10 con 0x7fdcf006d9d0 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd8003520 con 0x7fdcf006d9d0 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426994491 0 0) 0x7fdcd8003520 con 0x7fdcf006d9d0 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 >> v1:172.21.15.117:6790/0 conn(0x7fdcf00af1d0 legacy=0x7fdcf00af640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 >> v1:172.21.15.173:6789/0 conn(0x7fdcf013e820 legacy=0x7fdcf013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.216 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 monclient: found mon.noname-a 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00ab770 con 0x7fdcf006d9d0 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdce0001690 con 0x7fdcf006d9d0 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: dump: 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.217 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcf52a8640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcf52a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdce0001860 con 0x7fdcf006d9d0 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcf52a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcf52a8640 4 set_mon_vals no callback set 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 1 -- 172.21.15.117:0/1820459235 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdce0001cb0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.218 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.217+0000 7fdcedffb640 10 monclient: dump: 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.218+0000 7fdcf52a8640 10 monclient: shutdown 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.218+0000 7fdcf52a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:02.219 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.218+0000 7fdcf52a8640 1 -- 172.21.15.117:0/1820459235 >> v1:172.21.15.117:6789/0 conn(0x7fdcf006d9d0 legacy=0x7fdcf00b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.218+0000 7fdcf52a8640 1 -- 172.21.15.117:0/1820459235 shutdown_connections 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.218+0000 7fdcf52a8640 1 -- 172.21.15.117:0/1820459235 >> 172.21.15.117:0/1820459235 conn(0x7fdcf006b660 msgr2=0x7fdcf006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.219+0000 7fdcf52a8640 1 -- 172.21.15.117:0/1820459235 shutdown_connections 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.219+0000 7fdcf52a8640 1 -- 172.21.15.117:0/1820459235 wait complete. 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.220+0000 7fdcf52a8640 10 monclient: build_initial_monmap 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.220+0000 7fdcf52a8640 10 monclient: monmap: 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:02.220 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 librados: starting msgr at 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 librados: starting objecter 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 Processor -- start 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 -- start start 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 librados: setting wanted keys 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 1 librados: calling monclient init 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.221+0000 7fdcf52a8640 10 monclient: init 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf006d9d0 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf00af1d0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf0071000 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf006fda0 con 0x7fdcf0071000 2024-03-29T11:29:02.221 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf006efd0 con 0x7fdcf00af1d0 2024-03-29T11:29:02.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf0070080 con 0x7fdcf006d9d0 2024-03-29T11:29:02.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:02.222 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.222+0000 7fdcf52a8640 10 monclient(hunting): authenticate will time out at 1410.531797s 2024-03-29T11:29:02.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.225+0000 7fdceeffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fdcf006d9d0 0x7fdcf006ddb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:44581/0 (socket says 172.21.15.117:44581) 2024-03-29T11:29:02.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.225+0000 7fdceeffd640 1 -- 172.21.15.117:0/3158988985 learned_addr learned my addr 172.21.15.117:0/3158988985 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:02.225 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 client.?.objecter ms_handle_connect 0x7fdcf00af1d0 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 client.?.objecter ms_handle_connect 0x7fdcf006d9d0 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046604411 0 0) 0x7fdcf0070080 con 0x7fdcf006d9d0 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcdc004eb0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102886040 0 0) 0x7fdcf006efd0 con 0x7fdcf00af1d0 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.231+0000 7fdcd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:02.226 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0070080 con 0x7fdcf00af1d0 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864342556 0 0) 0x7fdcdc004eb0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf006efd0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdce0001b70 con 0x7fdcf006d9d0 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): dump: 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:02.227 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 629207928 0 0) 0x7fdcf0070080 con 0x7fdcf00af1d0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcdc004eb0 con 0x7fdcf00af1d0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1268385381 0 0) 0x7fdcf006efd0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.232+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 >> v1:172.21.15.173:6789/0 conn(0x7fdcf00af1d0 legacy=0x7fdcf00708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.233+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 >> v1:172.21.15.117:6789/0 conn(0x7fdcf0071000 legacy=0x7fdcf01e6920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.233+0000 7fdcd77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.233+0000 7fdcd77fe640 1 monclient: found mon.c 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.233+0000 7fdcd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.233+0000 7fdcd77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.234+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf0070360 con 0x7fdcf006d9d0 2024-03-29T11:29:02.228 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.234+0000 7fdcd77fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcd77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 10 monclient: _renew_subs 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf006e860 con 0x7fdcf006d9d0 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 10 monclient: _renew_subs 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf0071410 con 0x7fdcf006d9d0 2024-03-29T11:29:02.230 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.236+0000 7fdcf52a8640 1 librados: init done 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.237+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdce0004bf0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.237+0000 7fdcd77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.237+0000 7fdced7fa640 4 set_mon_vals no callback set 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.237+0000 7fdcf52a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fdce0001cd0 con 0x7fdcf006d9d0 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcd77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcd77fe640 10 monclient: dump: 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:02.231 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcf52a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcf52a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.238+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0005080 con 0x7fdcf006d9d0 2024-03-29T11:29:02.232 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.239+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fdce001ad70 con 0x7fdcf006d9d0 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.240+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1025824845 0 0) 0x7fdce0084b20 con 0x7fdcf006d9d0 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.240+0000 7fdcd77fe640 10 client.4673.objecter ms_dispatch 0x7fdcf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.240+0000 7fdcd77fe640 3 client.4673.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.240+0000 7fdcd77fe640 3 client.4673.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.240+0000 7fdcd77fe640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:29:02.233 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.241+0000 7fdcee7fc640 10 monclient: get_auth_request con 0x7fdcdc06de40 auth_method 0 2024-03-29T11:29:02.235 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.244+0000 7fdcd77fe640 10 client.4673.objecter ms_handle_connect 0x7fdcdc06de40 2024-03-29T11:29:02.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.247+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdce0050400 con 0x7fdcf006d9d0 2024-03-29T11:29:02.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.247+0000 7fdcd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:02.237 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.247+0000 7fdcd77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.565+0000 7fdcf52a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.565+0000 7fdcf52a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.565+0000 7fdcf52a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:02.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.565+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcd0005330 con 0x7fdcf006d9d0 2024-03-29T11:29:02.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.569+0000 7fdcd77fe640 1 -- 172.21.15.117:0/3158988985 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2639875723) 0x7fdce0052330 con 0x7fdcf006d9d0 2024-03-29T11:29:02.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.569+0000 7fdcd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:02.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.569+0000 7fdcd77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:02.560 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:02.560 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:19:06.394248+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:06.394248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 10 librados: watch_flush enter 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 10 librados: watch_flush exit 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 >> v1:172.21.15.117:6816/4276038175 conn(0x7fdcdc06de40 legacy=0x7fdcdc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 10 monclient: shutdown 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.572+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 >> v1:172.21.15.117:6790/0 conn(0x7fdcf006d9d0 legacy=0x7fdcf006ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.573+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 shutdown_connections 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.573+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 >> 172.21.15.117:0/3158988985 conn(0x7fdcf006b660 msgr2=0x7fdcf013ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.573+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 shutdown_connections 2024-03-29T11:29:02.564 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.573+0000 7fdcf52a8640 1 -- 172.21.15.117:0/3158988985 wait complete. 2024-03-29T11:29:02.565 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:02.573+0000 7fdcf52a8640 1 librados: shutdown 2024-03-29T11:29:02.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:09.387 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:09.539 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.548+0000 7fd2d0615640 10 monclient: get_monmap_and_config 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.548+0000 7fd2d0615640 10 monclient: build_initial_monmap 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.548+0000 7fd2d0615640 10 monclient: monmap: 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:09.549697+0000 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:09.549697+0000 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:09.540 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:09.541 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.549+0000 7fd2d0615640 1 Processor -- start 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.549+0000 7fd2d0615640 1 -- start start 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.549+0000 7fd2d0615640 10 monclient: init 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.550+0000 7fd2d0615640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.551+0000 7fd2d0615640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.551+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c80b0860 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:09.542 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.551+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c813ef30 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:09.545 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.551+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c80af8e0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2ce3b1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd2c80b0860 0x7fd2c813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:37985/0 (socket says 172.21.15.117:37985) 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2ce3b1640 1 -- 172.21.15.117:0/2640837755 learned_addr learned my addr 172.21.15.117:0/2640837755 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80b0c40 con 0x7fd2c80b0860 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80aed10 con 0x7fd2c80af8e0 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80afd50 con 0x7fd2c813ef30 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2d0615640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:09.546 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2d0615640 10 monclient(hunting): authenticate will time out at 1417.864568s 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060209208 0 0) 0x7fd2c80b0c40 con 0x7fd2c80b0860 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:09.547 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.555+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b4003520 con 0x7fd2c80b0860 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859439981 0 0) 0x7fd2c80aed10 con 0x7fd2c80af8e0 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c80b0c40 con 0x7fd2c80af8e0 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2853696982 0 0) 0x7fd2b4003520 con 0x7fd2c80b0860 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.559+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 >> v1:172.21.15.117:6790/0 conn(0x7fd2c813ef30 legacy=0x7fd2c80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 >> v1:172.21.15.173:6789/0 conn(0x7fd2c80af8e0 legacy=0x7fd2c8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.551 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: _finish_hunting 0 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 1 monclient: found mon.noname-a 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80ab1b0 con 0x7fd2c80b0860 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: _finish_auth 0 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: _check_auth_tickets 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd2c4001610 con 0x7fd2c80b0860 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.560+0000 7fd2cd3af640 10 monclient: dump: 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.552 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:09.553 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.562+0000 7fd2d0615640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.562+0000 7fd2d0615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.563+0000 7fd2cd3af640 1 -- 172.21.15.117:0/2640837755 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c4001770 con 0x7fd2c80b0860 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.563+0000 7fd2cd3af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.563+0000 7fd2d0615640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.563+0000 7fd2d0615640 4 set_mon_vals no callback set 2024-03-29T11:29:09.555 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.564+0000 7fd2d0615640 10 monclient: shutdown 2024-03-29T11:29:09.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.564+0000 7fd2d0615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:09.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.564+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 >> v1:172.21.15.117:6789/0 conn(0x7fd2c80b0860 legacy=0x7fd2c813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.565+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 shutdown_connections 2024-03-29T11:29:09.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.565+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 >> 172.21.15.117:0/2640837755 conn(0x7fd2c806b660 msgr2=0x7fd2c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:09.556 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.565+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 shutdown_connections 2024-03-29T11:29:09.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.565+0000 7fd2d0615640 1 -- 172.21.15.117:0/2640837755 wait complete. 2024-03-29T11:29:09.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.566+0000 7fd2d0615640 10 monclient: build_initial_monmap 2024-03-29T11:29:09.557 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.567+0000 7fd2d0615640 10 monclient: monmap: 2024-03-29T11:29:09.557 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:09.557 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:09.558 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.567+0000 7fd2d0615640 1 librados: starting msgr at 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.567+0000 7fd2d0615640 1 librados: starting objecter 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.568+0000 7fd2d0615640 1 Processor -- start 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.568+0000 7fd2d0615640 1 -- start start 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.568+0000 7fd2d0615640 1 librados: setting wanted keys 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.568+0000 7fd2d0615640 1 librados: calling monclient init 2024-03-29T11:29:09.559 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.568+0000 7fd2d0615640 10 monclient: init 2024-03-29T11:29:09.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.569+0000 7fd2d0615640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:09.560 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.569+0000 7fd2d0615640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:09.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c80af8e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:09.566 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c80b0860 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c813ef30 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c8137a60 con 0x7fd2c80af8e0 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80b1240 con 0x7fd2c80b0860 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c80b14b0 con 0x7fd2c813ef30 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2d0615640 10 monclient(hunting): authenticate will time out at 1417.881504s 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2ce3b1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd2c80af8e0 0x7fd2c81d63b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:41201/0 (socket says 172.21.15.117:41201) 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2ce3b1640 1 -- 172.21.15.117:0/2620730151 learned_addr learned my addr 172.21.15.117:0/2620730151 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2beffd640 10 client.?.objecter ms_handle_connect 0x7fd2c80af8e0 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.572+0000 7fd2beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958160587 0 0) 0x7fd2c8137a60 con 0x7fd2c80af8e0 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:09.567 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac004eb0 con 0x7fd2c80af8e0 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068487262 0 0) 0x7fd2ac004eb0 con 0x7fd2c80af8e0 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c8137a60 con 0x7fd2c80af8e0 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd2c4004a00 con 0x7fd2c80af8e0 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): dump: 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.568 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2742438450 0 0) 0x7fd2c8137a60 con 0x7fd2c80af8e0 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.573+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 >> v1:172.21.15.117:6790/0 conn(0x7fd2c813ef30 legacy=0x7fd2c81dd8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.575+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 >> v1:172.21.15.173:6789/0 conn(0x7fd2c80b0860 legacy=0x7fd2c81d6ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.578+0000 7fd2beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:09.569 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.578+0000 7fd2beffd640 1 monclient: found mon.a 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.578+0000 7fd2beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2beffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8136880 con 0x7fd2c80af8e0 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2beffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:29:09.570 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 10 monclient: _renew_subs 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c81373b0 con 0x7fd2c80af8e0 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 10 monclient: _renew_subs 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c81c8380 con 0x7fd2c80af8e0 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.579+0000 7fd2d0615640 1 librados: init done 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.580+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c4001a20 con 0x7fd2c80af8e0 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:09.571 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fd2c4005070 con 0x7fd2c80af8e0 2024-03-29T11:29:09.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:09.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:09.572 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2beffd640 10 monclient: dump: 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:09.573 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.581+0000 7fd2bffff640 4 set_mon_vals no callback set 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fd2c4005290 con 0x7fd2c80af8e0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2d0615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7fd2c4054c10 con 0x7fd2c80af8e0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2beffd640 10 client.4560.objecter ms_dispatch 0x7fd2c8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2beffd640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.582+0000 7fd2beffd640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2beffd640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2cdbb0640 10 monclient: get_auth_request con 0x7fd2ac06dba0 auth_method 0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2d0615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2d0615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b0005080 con 0x7fd2c80af8e0 2024-03-29T11:29:09.575 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.583+0000 7fd2beffd640 10 client.4560.objecter ms_handle_connect 0x7fd2ac06dba0 2024-03-29T11:29:09.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.591+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2c4050eb0 con 0x7fd2c80af8e0 2024-03-29T11:29:09.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.591+0000 7fd2beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:09.582 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.591+0000 7fd2beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:09.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.907+0000 7fd2d0615640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.907+0000 7fd2d0615640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.907+0000 7fd2d0615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.907+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2b0005330 con 0x7fd2c80af8e0 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.908+0000 7fd2beffd640 1 -- 172.21.15.117:0/2620730151 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7fd2c4051050 con 0x7fd2c80af8e0 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.908+0000 7fd2beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:09.899 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.908+0000 7fd2beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:09.901 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.912+0000 7fd2d0615640 10 librados: watch_flush enter 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 10 librados: watch_flush exit 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 >> v1:172.21.15.117:6816/4276038175 conn(0x7fd2ac06dba0 legacy=0x7fd2ac070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 10 monclient: shutdown 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:09.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.913+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 >> v1:172.21.15.117:6789/0 conn(0x7fd2c80af8e0 legacy=0x7fd2c81d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:09.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.915+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 shutdown_connections 2024-03-29T11:29:09.906 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.915+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 >> 172.21.15.117:0/2620730151 conn(0x7fd2c806b660 msgr2=0x7fd2c80ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:09.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.916+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 shutdown_connections 2024-03-29T11:29:09.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.916+0000 7fd2d0615640 1 -- 172.21.15.117:0/2620730151 wait complete. 2024-03-29T11:29:09.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:09.916+0000 7fd2d0615640 1 librados: shutdown 2024-03-29T11:29:09.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:16.736 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.893+0000 7eff17fff640 10 monclient: get_monmap_and_config 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.893+0000 7eff17fff640 10 monclient: build_initial_monmap 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.893+0000 7eff17fff640 10 monclient: monmap: 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:16.894974+0000 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:16.894974+0000 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:16.885 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.894+0000 7eff17fff640 1 Processor -- start 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.894+0000 7eff17fff640 1 -- start start 2024-03-29T11:29:16.886 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.894+0000 7eff17fff640 10 monclient: init 2024-03-29T11:29:16.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.896+0000 7eff17fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:16.887 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.896+0000 7eff17fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:16.888 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.897+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-a con 0x7eff100389e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:16.889 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.897+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-b con 0x7eff10038dc0 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff167fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7eff10038dc0 0x7eff100cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:53418/0 (socket says 172.21.15.117:53418) 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-c con 0x7eff10030660 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100c54c0 con 0x7eff100389e0 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff10036430 con 0x7eff10030660 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff10039840 con 0x7eff10038dc0 2024-03-29T11:29:16.892 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff17fff640 10 monclient(hunting): authenticate will time out at 1425.207492s 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.898+0000 7eff167fc640 1 -- 172.21.15.117:0/1694008258 learned_addr learned my addr 172.21.15.117:0/1694008258 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011625580 0 0) 0x7eff10039840 con 0x7eff10038dc0 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.899+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff00003520 con 0x7eff10038dc0 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 626493606 0 0) 0x7eff00003520 con 0x7eff10038dc0 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 >> v1:172.21.15.173:6789/0 conn(0x7eff10030660 legacy=0x7eff100d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 >> v1:172.21.15.117:6789/0 conn(0x7eff100389e0 legacy=0x7eff10033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:16.893 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10035690 con 0x7eff10038dc0 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7eff0c001690 con 0x7eff10038dc0 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.901+0000 7eff15ffb640 10 monclient: dump: 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:16.894 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff0c001920 con 0x7eff10038dc0 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 1 -- 172.21.15.117:0/1694008258 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7eff0c001cb0 con 0x7eff10038dc0 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:16.895 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.902+0000 7eff15ffb640 10 monclient: dump: 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:16.896 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.903+0000 7eff17fff640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.903+0000 7eff17fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.903+0000 7eff17fff640 4 set_mon_vals no callback set 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.904+0000 7eff17fff640 10 monclient: shutdown 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.904+0000 7eff17fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:16.897 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.904+0000 7eff17fff640 1 -- 172.21.15.117:0/1694008258 >> v1:172.21.15.117:6790/0 conn(0x7eff10038dc0 legacy=0x7eff100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.906+0000 7eff17fff640 1 -- 172.21.15.117:0/1694008258 shutdown_connections 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.906+0000 7eff17fff640 1 -- 172.21.15.117:0/1694008258 >> 172.21.15.117:0/1694008258 conn(0x7eff10005c00 msgr2=0x7eff10006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.906+0000 7eff17fff640 1 -- 172.21.15.117:0/1694008258 shutdown_connections 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.906+0000 7eff17fff640 1 -- 172.21.15.117:0/1694008258 wait complete. 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.907+0000 7eff17fff640 10 monclient: build_initial_monmap 2024-03-29T11:29:16.898 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.907+0000 7eff17fff640 10 monclient: monmap: 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:16.899 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 librados: starting msgr at 2024-03-29T11:29:16.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 librados: starting objecter 2024-03-29T11:29:16.900 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 Processor -- start 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 -- start start 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 librados: setting wanted keys 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 1 librados: calling monclient init 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.908+0000 7eff17fff640 10 monclient: init 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-c con 0x7eff10030660 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-a con 0x7eff100389e0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient(hunting): picked mon.noname-b con 0x7eff100c7910 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1002ff20 con 0x7eff100389e0 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1002f380 con 0x7eff100c7910 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff10030120 con 0x7eff10030660 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff17fff640 10 monclient(hunting): authenticate will time out at 1425.218450s 2024-03-29T11:29:16.901 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff16ffd640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7eff10030660 0x7eff10164150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:55632/0 (socket says 172.21.15.117:55632) 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7eff16ffd640 1 -- 172.21.15.117:0/18602510 learned_addr learned my addr 172.21.15.117:0/18602510 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7efeff7fe640 10 client.?.objecter ms_handle_connect 0x7eff10030660 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.909+0000 7efeff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955888506 0 0) 0x7eff10030120 con 0x7eff10030660 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:29:16.902 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04004eb0 con 0x7eff10030660 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.911+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920590216 0 0) 0x7eff04004eb0 con 0x7eff10030660 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff10030120 con 0x7eff10030660 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7eff08001990 con 0x7eff10030660 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:16.903 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.904 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1360142313 0 0) 0x7eff10030120 con 0x7eff10030660 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.912+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 >> v1:172.21.15.173:6789/0 conn(0x7eff100c7910 legacy=0x7eff1002e840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.913+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 >> v1:172.21.15.117:6789/0 conn(0x7eff100389e0 legacy=0x7eff1002f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 1 monclient: found mon.c 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1002ef50 con 0x7eff10030660 2024-03-29T11:29:16.907 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 client.?.objecter ms_handle_connect 0x7eff100389e0 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 10 monclient: _renew_subs 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff100c1e80 con 0x7eff10030660 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 10 client.4682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 10 monclient: _renew_subs 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff100c2400 con 0x7eff10030660 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7eff17fff640 1 librados: init done 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.914+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff08001af0 con 0x7eff10030660 2024-03-29T11:29:16.908 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.915+0000 7efeff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.915+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7eff08003a60 con 0x7eff10030660 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.915+0000 7efeff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.915+0000 7eff157fa640 4 set_mon_vals no callback set 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.916+0000 7efeff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.916+0000 7efeff7fe640 10 monclient: dump: 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:16.909 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:16.910 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:16.910 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:16.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.917+0000 7efefd7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:16.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.917+0000 7efefd7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:16.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.917+0000 7efefd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:16.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.917+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef8005080 con 0x7eff10030660 2024-03-29T11:29:16.911 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.917+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7eff08003bc0 con 0x7eff10030660 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.918+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7eff08084af0 con 0x7eff10030660 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.918+0000 7efeff7fe640 10 client.4682.objecter ms_dispatch 0x7eff18000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.918+0000 7efeff7fe640 3 client.4682.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.918+0000 7efeff7fe640 3 client.4682.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.918+0000 7efeff7fe640 20 client.4682.objecter dump_active .. 0 homeless 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.920+0000 7eff167fc640 10 monclient: get_auth_request con 0x7eff0406db60 auth_method 0 2024-03-29T11:29:16.912 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.920+0000 7efeff7fe640 10 client.4682.objecter ms_handle_connect 0x7eff0406db60 2024-03-29T11:29:16.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.929+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff080503d0 con 0x7eff10030660 2024-03-29T11:29:16.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.929+0000 7efeff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:16.920 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:16.929+0000 7efeff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.269+0000 7efefd7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.269+0000 7efefd7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.269+0000 7efefd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.269+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef8005330 con 0x7eff10030660 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.270+0000 7efeff7fe640 1 -- 172.21.15.117:0/18602510 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7eff08052300 con 0x7eff10030660 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.270+0000 7efeff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:17.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.270+0000 7efeff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:17.262 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:17.262 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.273+0000 7efefd7fa640 10 librados: watch_flush enter 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 10 librados: watch_flush exit 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 20 client.4682.objecter shutdown clearing up homeless session... 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 10 client.4682.objecter successfully canceled tick 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 >> v1:172.21.15.117:6816/4276038175 conn(0x7eff0406db60 legacy=0x7eff04070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 10 monclient: shutdown 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:17.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 >> v1:172.21.15.117:6790/0 conn(0x7eff10030660 legacy=0x7eff10164150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:17.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 shutdown_connections 2024-03-29T11:29:17.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 >> 172.21.15.117:0/18602510 conn(0x7eff10005c00 msgr2=0x7eff10034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:17.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.274+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 shutdown_connections 2024-03-29T11:29:17.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.275+0000 7efefd7fa640 1 -- 172.21.15.117:0/18602510 wait complete. 2024-03-29T11:29:17.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:17.275+0000 7efefd7fa640 1 librados: shutdown 2024-03-29T11:29:17.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:24.092 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.248+0000 7f4ad6ccc640 10 monclient: get_monmap_and_config 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.248+0000 7f4ad6ccc640 10 monclient: build_initial_monmap 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.248+0000 7f4ad6ccc640 10 monclient: monmap: 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:24.249829+0000 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:24.249829+0000 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:24.240 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.249+0000 7f4ad6ccc640 1 Processor -- start 2024-03-29T11:29:24.241 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.250+0000 7f4ad6ccc640 1 -- start start 2024-03-29T11:29:24.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.250+0000 7f4ad6ccc640 10 monclient: init 2024-03-29T11:29:24.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:24.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:24.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac8036a60 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:24.242 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac80c5de0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad5cca640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f4ac8036a60 0x7f4ac80c56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:57432/0 (socket says 172.21.15.117:57432) 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac80c3070 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad5cca640 1 -- 172.21.15.117:0/206761322 learned_addr learned my addr 172.21.15.117:0/206761322 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac802cc60 con 0x7f4ac80c5de0 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.251+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac802e040 con 0x7f4ac80c3070 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac802e320 con 0x7f4ac8036a60 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad6ccc640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad6ccc640 10 monclient(hunting): authenticate will time out at 1432.561559s 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053606761 0 0) 0x7f4ac802e320 con 0x7f4ac8036a60 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.244 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:29:24.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.252+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4abc007930 con 0x7f4ac8036a60 2024-03-29T11:29:24.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.253+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2047891193 0 0) 0x7f4abc007930 con 0x7f4ac8036a60 2024-03-29T11:29:24.245 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.253+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 >> v1:172.21.15.173:6789/0 conn(0x7f4ac80c3070 legacy=0x7f4ac80d0fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:24.246 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.254+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 >> v1:172.21.15.117:6789/0 conn(0x7f4ac80c5de0 legacy=0x7f4ac80c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: _finish_hunting 0 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 1 monclient: found mon.noname-b 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac802c830 con 0x7f4ac8036a60 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: _finish_auth 0 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: _check_auth_tickets 2024-03-29T11:29:24.248 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4acc001690 con 0x7f4ac8036a60 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad4cc8640 10 monclient: dump: 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:24.249 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad6ccc640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:29:24.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.256+0000 7f4ad6ccc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:24.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4acc001860 con 0x7f4ac8036a60 2024-03-29T11:29:24.250 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 1 -- 172.21.15.117:0/206761322 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4acc001c70 con 0x7f4ac8036a60 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad4cc8640 10 monclient: dump: 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:24.251 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad6ccc640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.258+0000 7f4ad6ccc640 4 set_mon_vals no callback set 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 10 monclient: shutdown 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 >> v1:172.21.15.117:6790/0 conn(0x7f4ac8036a60 legacy=0x7f4ac80c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 shutdown_connections 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 >> 172.21.15.117:0/206761322 conn(0x7f4ac8005c00 msgr2=0x7f4ac8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:24.252 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.259+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 shutdown_connections 2024-03-29T11:29:24.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.260+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/206761322 wait complete. 2024-03-29T11:29:24.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 10 monclient: build_initial_monmap 2024-03-29T11:29:24.253 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 10 monclient: monmap: 2024-03-29T11:29:24.253 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 1 librados: starting msgr at 2024-03-29T11:29:24.254 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 1 librados: starting objecter 2024-03-29T11:29:24.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 1 Processor -- start 2024-03-29T11:29:24.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 1 -- start start 2024-03-29T11:29:24.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.261+0000 7f4ad6ccc640 1 librados: setting wanted keys 2024-03-29T11:29:24.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.262+0000 7f4ad6ccc640 1 librados: calling monclient init 2024-03-29T11:29:24.255 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.262+0000 7f4ad6ccc640 10 monclient: init 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.262+0000 7f4ad6ccc640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac8036a60 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac80c3070 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac80c8d50 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80bde10 con 0x7f4ac80c8d50 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80be080 con 0x7f4ac8036a60 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80baa30 con 0x7f4ac80c3070 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad6ccc640 10 monclient(hunting): authenticate will time out at 1432.572153s 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad54c9640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f4ac80c3070 0x7f4ac80b9c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:58168/0 (socket says 172.21.15.117:58168) 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.263+0000 7f4ad54c9640 1 -- 172.21.15.117:0/3375147844 learned_addr learned my addr 172.21.15.117:0/3375147844 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.264+0000 7f4ac67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac80c3070 2024-03-29T11:29:24.256 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.264+0000 7f4ac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.265+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894146282 0 0) 0x7f4ac80baa30 con 0x7f4ac80c3070 2024-03-29T11:29:24.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.265+0000 7f4ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.265+0000 7f4ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.257 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.265+0000 7f4ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.265+0000 7f4ac67fc640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:29:24.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.266+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab4004eb0 con 0x7f4ac80c3070 2024-03-29T11:29:24.258 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac80c8d50 2024-03-29T11:29:24.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 client.?.objecter ms_handle_connect 0x7f4ac8036a60 2024-03-29T11:29:24.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:24.259 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221470836 0 0) 0x7f4ac80bde10 con 0x7f4ac80c8d50 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.267+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac80baa30 con 0x7f4ac80c8d50 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1382126324 0 0) 0x7f4ab4004eb0 con 0x7f4ac80c3070 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac80bde10 con 0x7f4ac80c3070 2024-03-29T11:29:24.260 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4ac0001890 con 0x7f4ac80c3070 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient(hunting): dump: 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:24.261 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802477320 0 0) 0x7f4ac80baa30 con 0x7f4ac80c8d50 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ab4004eb0 con 0x7f4ac80c8d50 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3143018696 0 0) 0x7f4ac80bde10 con 0x7f4ac80c3070 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 >> v1:172.21.15.173:6789/0 conn(0x7f4ac8036a60 legacy=0x7f4ac8158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 >> v1:172.21.15.117:6789/0 conn(0x7f4ac80c8d50 legacy=0x7f4ac80ba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 monclient: found mon.c 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.262 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac81585b0 con 0x7f4ac80c3070 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 10 monclient: _renew_subs 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac80be360 con 0x7f4ac80c3070 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 10 monclient: _renew_subs 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.268+0000 7f4ad6ccc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.269+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac80bea60 con 0x7f4ac80c3070 2024-03-29T11:29:24.263 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.269+0000 7f4ad6ccc640 1 librados: init done 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.272+0000 7f4ad6ccc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.272+0000 7f4ad6ccc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ad6ccc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac80bedb0 con 0x7f4ac80c3070 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ac0001a60 con 0x7f4ac80c3070 2024-03-29T11:29:24.264 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:24.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac7fff640 4 set_mon_vals no callback set 2024-03-29T11:29:24.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f4ac00037b0 con 0x7f4ac80c3070 2024-03-29T11:29:24.265 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.273+0000 7f4ac67fc640 10 monclient: dump: 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:24.266 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f4ac00039d0 con 0x7f4ac80c3070 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7f4ac0084a60 con 0x7f4ac80c3070 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 10 client.4688.objecter ms_dispatch 0x7f4ad0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ad5cca640 10 monclient: get_auth_request con 0x7f4ab406ddf0 auth_method 0 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.274+0000 7f4ac67fc640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-29T11:29:24.267 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.275+0000 7f4ac67fc640 10 client.4688.objecter ms_handle_connect 0x7f4ab406ddf0 2024-03-29T11:29:24.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.282+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ac00503c0 con 0x7f4ac80c3070 2024-03-29T11:29:24.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.282+0000 7f4ac67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:24.274 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.282+0000 7f4ac67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.609+0000 7f4ad6ccc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.609+0000 7f4ad6ccc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.609+0000 7f4ad6ccc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:24.600 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.609+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ac8005120 con 0x7f4ac80c3070 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.610+0000 7f4ac67fc640 1 -- 172.21.15.117:0/3375147844 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7f4ac00522f0 con 0x7f4ac80c3070 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.610+0000 7f4ac67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.610+0000 7f4ac67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:24.602 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:24.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.614+0000 7f4ad6ccc640 10 librados: watch_flush enter 2024-03-29T11:29:24.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.614+0000 7f4ad6ccc640 10 librados: watch_flush exit 2024-03-29T11:29:24.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.614+0000 7f4ad6ccc640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-29T11:29:24.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.614+0000 7f4ad6ccc640 10 client.4688.objecter successfully canceled tick 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.615+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 >> v1:172.21.15.117:6816/4276038175 conn(0x7f4ab406ddf0 legacy=0x7f4ab40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.615+0000 7f4ad6ccc640 10 monclient: shutdown 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.615+0000 7f4ad6ccc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.615+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 >> v1:172.21.15.117:6790/0 conn(0x7f4ac80c3070 legacy=0x7f4ac80b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.616+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 shutdown_connections 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.616+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 >> 172.21.15.117:0/3375147844 conn(0x7f4ac8005c00 msgr2=0x7f4ac80bc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.616+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 shutdown_connections 2024-03-29T11:29:24.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.616+0000 7f4ad6ccc640 1 -- 172.21.15.117:0/3375147844 wait complete. 2024-03-29T11:29:24.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:24.617+0000 7f4ad6ccc640 1 librados: shutdown 2024-03-29T11:29:24.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:31.432 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.592+0000 7f2fd6979640 10 monclient: get_monmap_and_config 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.592+0000 7f2fd6979640 10 monclient: build_initial_monmap 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.592+0000 7f2fd6979640 10 monclient: monmap: 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.584 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:31.593731+0000 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:31.593731+0000 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.593+0000 7f2fd6979640 1 Processor -- start 2024-03-29T11:29:31.585 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.593+0000 7f2fd6979640 1 -- start start 2024-03-29T11:29:31.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.593+0000 7f2fd6979640 10 monclient: init 2024-03-29T11:29:31.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.594+0000 7f2fd6979640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:31.586 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.594+0000 7f2fd6979640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.594+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd00b4330 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd01470b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fcf7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f2fd01470b0 0x7f2fd00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:41696/0 (socket says 172.21.15.117:41696) 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fcffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f2fd00b4330 0x7f2fd00b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:59020/0 (socket says 172.21.15.117:59020) 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd4f16640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f2fd01439b0 0x7f2fd0151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:46108/0 (socket says 172.21.15.117:46108) 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fcf7fe640 1 -- 172.21.15.117:0/3077414328 learned_addr learned my addr 172.21.15.117:0/3077414328 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd01439b0 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd006f2a0 con 0x7f2fd00b4330 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0146b90 con 0x7f2fd01470b0 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00b2ca0 con 0x7f2fd01439b0 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:31.587 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.595+0000 7f2fd6979640 10 monclient(hunting): authenticate will time out at 1439.904679s 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126886340 0 0) 0x7f2fd0146b90 con 0x7f2fd01470b0 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb8003520 con 0x7f2fd01470b0 2024-03-29T11:29:31.588 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572835415 0 0) 0x7f2fd00b2ca0 con 0x7f2fd01439b0 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd0146b90 con 0x7f2fd01439b0 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450985768 0 0) 0x7f2fd006f2a0 con 0x7f2fd00b4330 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd00b2ca0 con 0x7f2fd00b4330 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.596+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674898428 0 0) 0x7f2fb8003520 con 0x7f2fd01470b0 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 >> v1:172.21.15.117:6790/0 conn(0x7f2fd01439b0 legacy=0x7f2fd0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 >> v1:172.21.15.117:6789/0 conn(0x7f2fd00b4330 legacy=0x7f2fd00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.589 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 monclient: found mon.noname-c 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd006e0c0 con 0x7f2fd01470b0 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2fc40015c0 con 0x7f2fd01470b0 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: dump: 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.590 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fd6979640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:29:31.591 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fd6979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc40017a0 con 0x7f2fd01470b0 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 1 -- 172.21.15.117:0/3077414328 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2fc4001bf0 con 0x7f2fd01470b0 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fceffd640 10 monclient: dump: 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.593 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fd6979640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.597+0000 7f2fd6979640 4 set_mon_vals no callback set 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.598+0000 7f2fd6979640 10 monclient: shutdown 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.598+0000 7f2fd6979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.598+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 >> v1:172.21.15.173:6789/0 conn(0x7f2fd01470b0 legacy=0x7f2fd00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.598+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 shutdown_connections 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.598+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 >> 172.21.15.117:0/3077414328 conn(0x7f2fd0066a00 msgr2=0x7f2fd0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.599+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 shutdown_connections 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.599+0000 7f2fd6979640 1 -- 172.21.15.117:0/3077414328 wait complete. 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.600+0000 7f2fd6979640 10 monclient: build_initial_monmap 2024-03-29T11:29:31.594 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.600+0000 7f2fd6979640 10 monclient: monmap: 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.600+0000 7f2fd6979640 1 librados: starting msgr at 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.600+0000 7f2fd6979640 1 librados: starting objecter 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.600+0000 7f2fd6979640 1 Processor -- start 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 1 -- start start 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 1 librados: setting wanted keys 2024-03-29T11:29:31.595 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 1 librados: calling monclient init 2024-03-29T11:29:31.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 10 monclient: init 2024-03-29T11:29:31.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:31.596 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.601+0000 7f2fd6979640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd00b4330 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd01439b0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd01470b0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0120100 con 0x7f2fd01470b0 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0140d30 con 0x7f2fd01439b0 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fcf7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f2fd01439b0 0x7f2fd013ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:42120/0 (socket says 172.21.15.117:42120) 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fcf7fe640 1 -- 172.21.15.117:0/3978400029 learned_addr learned my addr 172.21.15.117:0/3978400029 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0121530 con 0x7f2fd00b4330 2024-03-29T11:29:31.603 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 client.?.objecter ms_handle_connect 0x7f2fd01439b0 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fd6979640 10 monclient(hunting): authenticate will time out at 1439.911429s 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 client.?.objecter ms_handle_connect 0x7f2fd00b4330 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452437206 0 0) 0x7f2fd0140d30 con 0x7f2fd01439b0 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.602+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fbc004eb0 con 0x7f2fd01439b0 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083338338 0 0) 0x7f2fd0121530 con 0x7f2fd00b4330 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.604 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd0140d30 con 0x7f2fd00b4330 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 client.?.objecter ms_handle_connect 0x7f2fd01470b0 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355255991 0 0) 0x7f2fbc004eb0 con 0x7f2fd01439b0 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fd0121530 con 0x7f2fd01439b0 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2fc40047e0 con 0x7f2fd01439b0 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): dump: 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.605 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504194449 0 0) 0x7f2fd0140d30 con 0x7f2fd00b4330 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fbc004eb0 con 0x7f2fd00b4330 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2fc0001830 con 0x7f2fd00b4330 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.603+0000 7f2fccff9640 10 monclient(hunting): dump: 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.606 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206251369 0 0) 0x7f2fd0120100 con 0x7f2fd01470b0 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd0140d30 con 0x7f2fd01470b0 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191461901 0 0) 0x7f2fd0121530 con 0x7f2fd01439b0 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 >> v1:172.21.15.117:6790/0 conn(0x7f2fd00b4330 legacy=0x7f2fd01de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.607 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 >> v1:172.21.15.117:6789/0 conn(0x7f2fd01470b0 legacy=0x7f2fd0140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.604+0000 7f2fccff9640 1 monclient: found mon.b 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd01df020 con 0x7f2fd01439b0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 10 monclient: _renew_subs 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc4004c60 con 0x7f2fd01439b0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd01414c0 con 0x7f2fd01439b0 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:31.608 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 10 monclient: _renew_subs 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd01204a0 con 0x7f2fd01439b0 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fd6979640 1 librados: init done 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.605+0000 7f2fce7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.606+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f2fc4004ff0 con 0x7f2fd01439b0 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.606+0000 7f2fccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.606+0000 7f2fccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.606+0000 7f2fccff9640 10 monclient: dump: 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:31.609 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.607+0000 7f2fd6979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.608+0000 7f2fd6979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.608+0000 7f2fd6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.608+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb0005080 con 0x7f2fd01439b0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.608+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f2fc4005150 con 0x7f2fd01439b0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7f2fc4084c40 con 0x7f2fd01439b0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 10 client.4573.objecter ms_dispatch 0x7f2fd0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 3 client.4573.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 3 client.4573.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fcffff640 10 monclient: get_auth_request con 0x7f2fbc06df40 auth_method 0 2024-03-29T11:29:31.610 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:29:31.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.609+0000 7f2fccff9640 10 client.4573.objecter ms_handle_connect 0x7f2fbc06df40 2024-03-29T11:29:31.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.615+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fc4050520 con 0x7f2fd01439b0 2024-03-29T11:29:31.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.615+0000 7f2fccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:31.611 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:31.615+0000 7f2fccff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:32.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.032+0000 7f2fd6979640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.032+0000 7f2fd6979640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.032+0000 7f2fd6979640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:32.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.032+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb0005330 con 0x7f2fd01439b0 2024-03-29T11:29:32.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.034+0000 7f2fccff9640 1 -- 172.21.15.117:0/3978400029 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7f2fc4052450 con 0x7f2fd01439b0 2024-03-29T11:29:32.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.034+0000 7f2fccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:32.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.034+0000 7f2fccff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:32.026 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:32.026 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:32.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 10 librados: watch_flush enter 2024-03-29T11:29:32.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 10 librados: watch_flush exit 2024-03-29T11:29:32.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 >> v1:172.21.15.117:6816/4276038175 conn(0x7f2fbc06df40 legacy=0x7f2fbc070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 10 monclient: shutdown 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 >> v1:172.21.15.173:6789/0 conn(0x7f2fd01439b0 legacy=0x7f2fd013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 shutdown_connections 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.037+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 >> 172.21.15.117:0/3978400029 conn(0x7f2fd0066a00 msgr2=0x7f2fd0142e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.038+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 shutdown_connections 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.038+0000 7f2fd6979640 1 -- 172.21.15.117:0/3978400029 wait complete. 2024-03-29T11:29:32.030 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:32.038+0000 7f2fd6979640 1 librados: shutdown 2024-03-29T11:29:32.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:38.853 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:39.001 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.009+0000 7f8ddd5d6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.009+0000 7f8ddd5d6640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.010+0000 7f8ddd5d6640 10 monclient: monmap: 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:39.011094+0000 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:39.011094+0000 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:39.002 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.011+0000 7f8ddd5d6640 1 Processor -- start 2024-03-29T11:29:39.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.011+0000 7f8ddd5d6640 1 -- start start 2024-03-29T11:29:39.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.011+0000 7f8ddd5d6640 10 monclient: init 2024-03-29T11:29:39.006 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd813f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8dd80a9d70 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd80aaef0 con 0x7f8dd813f650 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8145bd0 con 0x7f8dd81457f0 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8145eb0 con 0x7f8dd80a9d70 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.012+0000 7f8ddd5d6640 10 monclient(hunting): authenticate will time out at 1447.321150s 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.015+0000 7f8dd77fe640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 0x7f8dd814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:34534/0 (socket says 172.21.15.117:34534) 2024-03-29T11:29:39.007 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.015+0000 7f8dd77fe640 1 -- 172.21.15.117:0/633092367 learned_addr learned my addr 172.21.15.117:0/633092367 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:39.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.015+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603467221 0 0) 0x7f8dd8145eb0 con 0x7f8dd80a9d70 2024-03-29T11:29:39.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.016+0000 7f8dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.016+0000 7f8dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.008 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.016+0000 7f8dd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.016+0000 7f8dd5ffb640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:29:39.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.016+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc0003520 con 0x7f8dd80a9d70 2024-03-29T11:29:39.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.017+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1821516256 0 0) 0x7f8dc0003520 con 0x7f8dd80a9d70 2024-03-29T11:29:39.009 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.017+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.173:6789/0 conn(0x7f8dd81457f0 legacy=0x7f8dd813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6789/0 conn(0x7f8dd813f650 legacy=0x7f8dd80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:39.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.011 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8146340 con 0x7f8dd80a9d70 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8dc8001690 con 0x7f8dd80a9d70 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.018+0000 7f8dd5ffb640 10 monclient: dump: 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.012 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:39.013 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:39.013 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:39.013 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.020+0000 7f8ddd5d6640 5 monclient: authenticate success, global_id 4697 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.020+0000 7f8ddd5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd77fe640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 legacy=0x7f8dd814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd77fe640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 legacy=0x7f8dd814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd77fe640 1 --1- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 0x7f8dd814d8c0 :-1 s=OPENED pgs=1031571 cs=1 l=1).handle_message read tag failed 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd77fe640 1 --1- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 0x7f8dd814d8c0 :-1 s=OPENED pgs=1031571 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.117:6790/0 2024-03-29T11:29:39.014 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dd80a9d70 legacy=0x7f8dd814d8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient(hunting): picked mon.b con 0x7f8dd81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient(hunting): picked mon.a con 0x7f8dd813f650 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient(hunting): picked mon.c con 0x7f8dc000b090 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8dc0003520 con 0x7f8dd813f650 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8dd80aaef0 con 0x7f8dd81457f0 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8dd8145bd0 con 0x7f8dc000b090 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.021+0000 7f8dd5ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.117:6790/0 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.022+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912883355 0 0) 0x7f8dd8145bd0 con 0x7f8dc000b090 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.022+0000 7f8dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.022+0000 7f8dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.022+0000 7f8dd5ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:29:39.015 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.022+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f8dc0010e10 con 0x7f8dc000b090 2024-03-29T11:29:39.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.024+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2673986175 0 0) 0x7f8dc0010e10 con 0x7f8dc000b090 2024-03-29T11:29:39.016 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.024+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.173:6789/0 conn(0x7f8dd81457f0 legacy=0x7f8dc0003aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.025+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6789/0 conn(0x7f8dd813f650 legacy=0x7f8dc0009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 1 monclient: found mon.c 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f8dd8145eb0 con 0x7f8dc000b090 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.018 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8dc80048a0 con 0x7f8dc000b090 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: dump: 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:39.019 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8ddd5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:39.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 1 -- 172.21.15.117:0/633092367 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dc8004ac0 con 0x7f8dc000b090 2024-03-29T11:29:39.020 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8dd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8ddd5d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.026+0000 7f8ddd5d6640 4 set_mon_vals no callback set 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.027+0000 7f8ddd5d6640 10 monclient: shutdown 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.027+0000 7f8ddd5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.027+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/633092367 >> v1:172.21.15.117:6790/0 conn(0x7f8dc000b090 legacy=0x7f8dc000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.028+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/633092367 shutdown_connections 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.028+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/633092367 >> 172.21.15.117:0/633092367 conn(0x7f8dd8066a00 msgr2=0x7f8dd8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.029+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/633092367 shutdown_connections 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.029+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/633092367 wait complete. 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.030+0000 7f8ddd5d6640 10 monclient: build_initial_monmap 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.030+0000 7f8ddd5d6640 10 monclient: monmap: 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:39.022 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.023 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:39.023 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:39.023 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:39.023 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 librados: starting msgr at 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 librados: starting objecter 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 Processor -- start 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 -- start start 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 librados: setting wanted keys 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 1 librados: calling monclient init 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.031+0000 7f8ddd5d6640 10 monclient: init 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.032+0000 7f8ddd5d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.032+0000 7f8ddd5d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.032+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-b con 0x7f8dc000b090 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:39.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.032+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-c con 0x7f8dd8139180 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:39.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.032+0000 7f8ddd5d6640 10 monclient(hunting): picked mon.noname-a con 0x7f8dd8137550 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:39.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.036+0000 7f8dd67fc640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7f8dd8139180 0x7f8dd81db100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:38216/0 (socket says 172.21.15.117:38216) 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.036+0000 7f8dd67fc640 1 -- 172.21.15.117:0/3673525431 learned_addr learned my addr 172.21.15.117:0/3673525431 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.036+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd813a000 con 0x7f8dd8137550 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.036+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8139c60 con 0x7f8dc000b090 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.036+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd8145210 con 0x7f8dd8139180 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.037+0000 7f8ddd5d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.037+0000 7f8ddd5d6640 10 monclient(hunting): authenticate will time out at 1447.346125s 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.037+0000 7f8dbf7fe640 10 client.?.objecter ms_handle_connect 0x7f8dd8137550 2024-03-29T11:29:39.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.037+0000 7f8dbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.037+0000 7f8dbf7fe640 10 client.?.objecter ms_handle_connect 0x7f8dc000b090 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956224327 0 0) 0x7f8dd813a000 con 0x7f8dd8137550 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc4004eb0 con 0x7f8dd8137550 2024-03-29T11:29:39.033 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 client.?.objecter ms_handle_connect 0x7f8dd8139180 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127297535 0 0) 0x7f8dd8139c60 con 0x7f8dc000b090 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.038+0000 7f8dbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd813a000 con 0x7f8dc000b090 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756103200 0 0) 0x7f8dd813a000 con 0x7f8dc000b090 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd8139c60 con 0x7f8dc000b090 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.039+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8dcc001830 con 0x7f8dc000b090 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:39.034 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838512625 0 0) 0x7f8dd8145210 con 0x7f8dd8139180 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:39.035 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 10 monclient(hunting): my global_id is 4700 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.040+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd813a000 con 0x7f8dd8139180 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2953305797 0 0) 0x7f8dd8139c60 con 0x7f8dc000b090 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 >> v1:172.21.15.117:6790/0 conn(0x7f8dd8139180 legacy=0x7f8dd81db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 >> v1:172.21.15.117:6789/0 conn(0x7f8dd8137550 legacy=0x7f8dd81e69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 1 monclient: found mon.b 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.041+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd81356c0 con 0x7f8dc000b090 2024-03-29T11:29:39.037 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:39.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:39.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:39.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dcc001d40 con 0x7f8dc000b090 2024-03-29T11:29:39.038 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f8dcc0037b0 con 0x7f8dc000b090 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dbf7fe640 10 monclient: dump: 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:39.039 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8dd57fa640 4 set_mon_vals no callback set 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 5 monclient: authenticate success, global_id 4579 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 10 monclient: _renew_subs 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd81db810 con 0x7f8dc000b090 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 10 monclient: _renew_subs 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd81dbd40 con 0x7f8dc000b090 2024-03-29T11:29:39.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.042+0000 7f8ddd5d6640 1 librados: init done 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.047+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f8dcc019780 con 0x7f8dc000b090 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.048+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7f8dcc083710 con 0x7f8dc000b090 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.048+0000 7f8dbf7fe640 10 client.4579.objecter ms_dispatch 0x7f8dd8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.048+0000 7f8dbf7fe640 3 client.4579.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.048+0000 7f8dbf7fe640 3 client.4579.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:39.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.048+0000 7f8dbf7fe640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8dd67fc640 10 monclient: get_auth_request con 0x7f8dc406db10 auth_method 0 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8ddd5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8ddd5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8ddd5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd813b340 con 0x7f8dc000b090 2024-03-29T11:29:39.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.050+0000 7f8dbf7fe640 10 client.4579.objecter ms_handle_connect 0x7f8dc406db10 2024-03-29T11:29:39.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.058+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dcc04ee20 con 0x7f8dc000b090 2024-03-29T11:29:39.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.058+0000 7f8dbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:39.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.058+0000 7f8dbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:39.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.436+0000 7f8ddd5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.436+0000 7f8ddd5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.436+0000 7f8ddd5d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T11:29:39.428 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.436+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd8139df0 con 0x7f8dc000b090 2024-03-29T11:29:39.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.437+0000 7f8dbf7fe640 1 -- 172.21.15.117:0/3673525431 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7f8dcc050f20 con 0x7f8dc000b090 2024-03-29T11:29:39.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.437+0000 7f8dbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:39.429 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.437+0000 7f8dbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:39.429 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:39.429 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:39.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 10 librados: watch_flush enter 2024-03-29T11:29:39.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 10 librados: watch_flush exit 2024-03-29T11:29:39.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-29T11:29:39.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 10 client.4579.objecter successfully canceled tick 2024-03-29T11:29:39.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 >> v1:172.21.15.117:6816/4276038175 conn(0x7f8dc406db10 legacy=0x7f8dc406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 10 monclient: shutdown 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.442+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 >> v1:172.21.15.173:6789/0 conn(0x7f8dc000b090 legacy=0x7f8dd8134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.443+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 shutdown_connections 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.443+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 >> 172.21.15.117:0/3673525431 conn(0x7f8dd8066a00 msgr2=0x7f8dd80b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:39.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.444+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 shutdown_connections 2024-03-29T11:29:39.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.444+0000 7f8ddd5d6640 1 -- 172.21.15.117:0/3673525431 wait complete. 2024-03-29T11:29:39.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:39.444+0000 7f8ddd5d6640 1 librados: shutdown 2024-03-29T11:29:39.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:46.263 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:46.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.416+0000 7f7b4a02b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:46.408 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.416+0000 7f7b4a02b640 10 monclient: build_initial_monmap 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.416+0000 7f7b4a02b640 10 monclient: monmap: 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:46.417903+0000 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:46.417903+0000 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:46.409 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.417+0000 7f7b4a02b640 1 Processor -- start 2024-03-29T11:29:46.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 1 -- start start 2024-03-29T11:29:46.410 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient: init 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b44149a90 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b44149e70 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b440af740 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4414a3d0 con 0x7f7b44149e70 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44072730 con 0x7f7b44149a90 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44071f90 con 0x7f7b440af740 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.418+0000 7f7b4a02b640 10 monclient(hunting): authenticate will time out at 1454.727983s 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.419+0000 7f7b48828640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7b44149e70 0x7f7b4406be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:36536/0 (socket says 172.21.15.117:36536) 2024-03-29T11:29:46.411 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.419+0000 7f7b48828640 1 -- 172.21.15.117:0/3533528180 learned_addr learned my addr 172.21.15.117:0/3533528180 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142217386 0 0) 0x7f7b4414a3d0 con 0x7f7b44149e70 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30003520 con 0x7f7b44149e70 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378479794 0 0) 0x7f7b30003520 con 0x7f7b44149e70 2024-03-29T11:29:46.414 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.420+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 >> v1:172.21.15.117:6790/0 conn(0x7f7b440af740 legacy=0x7f7b4406c5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:46.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 >> v1:172.21.15.173:6789/0 conn(0x7f7b44149a90 legacy=0x7f7b44071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:46.417 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 monclient: found mon.noname-a 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44072120 con 0x7f7b44149e70 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7b34001690 con 0x7f7b44149e70 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: dump: 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:46.418 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 <== mon.0 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b340018b0 con 0x7f7b44149e70 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 1 -- 172.21.15.117:0/3533528180 <== mon.0 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7b34001c80 con 0x7f7b44149e70 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.423+0000 7f7b3bfff640 10 monclient: dump: 2024-03-29T11:29:46.419 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.424+0000 7f7b4a02b640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.424+0000 7f7b4a02b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.424+0000 7f7b4a02b640 4 set_mon_vals no callback set 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 10 monclient: shutdown 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:46.420 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3533528180 >> v1:172.21.15.117:6789/0 conn(0x7f7b44149e70 legacy=0x7f7b4406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3533528180 shutdown_connections 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3533528180 >> 172.21.15.117:0/3533528180 conn(0x7f7b440671c0 msgr2=0x7f7b440b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.425+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3533528180 shutdown_connections 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.426+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3533528180 wait complete. 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.426+0000 7f7b4a02b640 10 monclient: build_initial_monmap 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.426+0000 7f7b4a02b640 10 monclient: monmap: 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:46.421 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 librados: starting msgr at 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 librados: starting objecter 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 Processor -- start 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 -- start start 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 librados: setting wanted keys 2024-03-29T11:29:46.422 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 1 librados: calling monclient init 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.427+0000 7f7b4a02b640 10 monclient: init 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b440af740 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b44149a90 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b44149e70 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:46.423 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44135750 con 0x7f7b44149a90 2024-03-29T11:29:46.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441358e0 con 0x7f7b44149e70 2024-03-29T11:29:46.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441349f0 con 0x7f7b440af740 2024-03-29T11:29:46.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:46.424 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.428+0000 7f7b4a02b640 10 monclient(hunting): authenticate will time out at 1454.737337s 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.432+0000 7f7b48828640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7b44149a90 0x7f7b441d7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:44512/0 (socket says 172.21.15.117:44512) 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.432+0000 7f7b48828640 1 -- 172.21.15.117:0/3484229561 learned_addr learned my addr 172.21.15.117:0/3484229561 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 client.?.objecter ms_handle_connect 0x7f7b44149a90 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 client.?.objecter ms_handle_connect 0x7f7b44149e70 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841619275 0 0) 0x7f7b44135750 con 0x7f7b44149a90 2024-03-29T11:29:46.430 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b28004eb0 con 0x7f7b44149a90 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106461889 0 0) 0x7f7b441358e0 con 0x7f7b44149e70 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b44135750 con 0x7f7b44149e70 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436657882 0 0) 0x7f7b28004eb0 con 0x7f7b44149a90 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b441358e0 con 0x7f7b44149a90 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7b34001610 con 0x7f7b44149a90 2024-03-29T11:29:46.431 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.433+0000 7f7b39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3374465582 0 0) 0x7f7b441358e0 con 0x7f7b44149a90 2024-03-29T11:29:46.432 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 >> v1:172.21.15.117:6790/0 conn(0x7f7b440af740 legacy=0x7f7b441dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 >> v1:172.21.15.173:6789/0 conn(0x7f7b44149e70 legacy=0x7f7b44135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 monclient: found mon.a 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44134b80 con 0x7f7b44149a90 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:46.433 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 10 monclient: _renew_subs 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b441db410 con 0x7f7b44149a90 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 10 monclient: _renew_subs 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44133520 con 0x7f7b44149a90 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b4a02b640 1 librados: init done 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b34004a90 con 0x7f7b44149a90 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:46.434 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7f7b34004da0 con 0x7f7b44149a90 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b39ffb640 10 monclient: dump: 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.434+0000 7f7b3b7fe640 4 set_mon_vals no callback set 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.436+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7f7b34004f00 con 0x7f7b44149a90 2024-03-29T11:29:46.435 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.436+0000 7f7b4a02b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.436+0000 7f7b4a02b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.436+0000 7f7b4a02b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.436+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b24005080 con 0x7f7b44149a90 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b49029640 10 monclient: get_auth_request con 0x7f7b2806dd40 auth_method 0 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b39ffb640 10 client.4587.objecter ms_handle_connect 0x7f7b2806dd40 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7f7b34084be0 con 0x7f7b44149a90 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b39ffb640 10 client.4587.objecter ms_dispatch 0x7f7b44000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b39ffb640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.437+0000 7f7b39ffb640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:46.436 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.438+0000 7f7b39ffb640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:29:46.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.445+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b340504c0 con 0x7f7b44149a90 2024-03-29T11:29:46.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.445+0000 7f7b39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:46.439 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.445+0000 7f7b39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:46.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.771+0000 7f7b4a02b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:46.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.771+0000 7f7b4a02b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:46.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.771+0000 7f7b4a02b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:29:46.763 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.771+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b24005600 con 0x7f7b44149a90 2024-03-29T11:29:46.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.772+0000 7f7b39ffb640 1 -- 172.21.15.117:0/3484229561 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7f7b340523f0 con 0x7f7b44149a90 2024-03-29T11:29:46.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.772+0000 7f7b39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:46.764 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.772+0000 7f7b39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:46.764 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:46.764 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:46.767 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.775+0000 7f7b4a02b640 10 librados: watch_flush enter 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.775+0000 7f7b4a02b640 10 librados: watch_flush exit 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.775+0000 7f7b4a02b640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.775+0000 7f7b4a02b640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.776+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 >> v1:172.21.15.117:6816/4276038175 conn(0x7f7b2806dd40 legacy=0x7f7b28070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.776+0000 7f7b4a02b640 10 monclient: shutdown 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.776+0000 7f7b4a02b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:46.768 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.776+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 >> v1:172.21.15.117:6789/0 conn(0x7f7b44149a90 legacy=0x7f7b441d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:46.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.777+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 shutdown_connections 2024-03-29T11:29:46.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.777+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 >> 172.21.15.117:0/3484229561 conn(0x7f7b440671c0 msgr2=0x7f7b4406bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:46.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.777+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 shutdown_connections 2024-03-29T11:29:46.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.777+0000 7f7b4a02b640 1 -- 172.21.15.117:0/3484229561 wait complete. 2024-03-29T11:29:46.769 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:46.777+0000 7f7b4a02b640 1 librados: shutdown 2024-03-29T11:29:46.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:53.593 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:53.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.728+0000 7fe0a4081640 10 monclient: get_monmap_and_config 2024-03-29T11:29:53.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.728+0000 7fe0a4081640 10 monclient: build_initial_monmap 2024-03-29T11:29:53.720 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.728+0000 7fe0a4081640 10 monclient: monmap: 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:29:53.729571+0000 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:29:53.729571+0000 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:29:53.721 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.729+0000 7fe0a4081640 1 Processor -- start 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.729+0000 7fe0a4081640 1 -- start start 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.729+0000 7fe0a4081640 10 monclient: init 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-a con 0x7fe09c1457f0 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-b con 0x7fe09c13f650 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:53.723 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a161c640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe09c13f650 0x7fe09c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:39330/0 (socket says 172.21.15.117:39330) 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-c con 0x7fe09c0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c0aaef0 con 0x7fe09c1457f0 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c145bd0 con 0x7fe09c0a9d70 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c145eb0 con 0x7fe09c13f650 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a4081640 10 monclient(hunting): authenticate will time out at 1462.039373s 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a161c640 1 -- 172.21.15.117:0/2770212717 learned_addr learned my addr 172.21.15.117:0/2770212717 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123680383 0 0) 0x7fe09c145eb0 con 0x7fe09c13f650 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 10 monclient(hunting): _negotiate 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:29:53.724 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.730+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088003520 con 0x7fe09c13f650 2024-03-29T11:29:53.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.731+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709739873 0 0) 0x7fe088003520 con 0x7fe09c13f650 2024-03-29T11:29:53.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.731+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 >> v1:172.21.15.173:6789/0 conn(0x7fe09c0a9d70 legacy=0x7fe09c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:53.725 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.731+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 >> v1:172.21.15.117:6789/0 conn(0x7fe09c1457f0 legacy=0x7fe09c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: _finish_hunting 0 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 1 monclient: found mon.noname-b 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c146340 con 0x7fe09c13f650 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: _finish_auth 0 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: _check_auth_tickets 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:53.728 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe08c001690 con 0x7fe09c13f650 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: dump: 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.729 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a4081640 5 monclient: authenticate success, global_id 4703 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a4081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe08c001920 con 0x7fe09c13f650 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 1 -- 172.21.15.117:0/2770212717 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe08c001cb0 con 0x7fe09c13f650 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a0e1b640 10 monclient: dump: 2024-03-29T11:29:53.730 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a4081640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.734+0000 7fe0a4081640 4 set_mon_vals no callback set 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.735+0000 7fe0a4081640 10 monclient: shutdown 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.735+0000 7fe0a4081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.735+0000 7fe0a4081640 1 -- 172.21.15.117:0/2770212717 >> v1:172.21.15.117:6790/0 conn(0x7fe09c13f650 legacy=0x7fe09c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:53.731 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.736+0000 7fe0a4081640 1 -- 172.21.15.117:0/2770212717 shutdown_connections 2024-03-29T11:29:53.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.736+0000 7fe0a4081640 1 -- 172.21.15.117:0/2770212717 >> 172.21.15.117:0/2770212717 conn(0x7fe09c066a00 msgr2=0x7fe09c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:53.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.736+0000 7fe0a4081640 1 -- 172.21.15.117:0/2770212717 shutdown_connections 2024-03-29T11:29:53.732 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.736+0000 7fe0a4081640 1 -- 172.21.15.117:0/2770212717 wait complete. 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.737+0000 7fe0a4081640 10 monclient: build_initial_monmap 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.737+0000 7fe0a4081640 10 monclient: monmap: 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:29:53.735 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 librados: starting msgr at 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 librados: starting objecter 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 Processor -- start 2024-03-29T11:29:53.736 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 -- start start 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 librados: setting wanted keys 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 1 librados: calling monclient init 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.738+0000 7fe0a4081640 10 monclient: init 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.739+0000 7fe0a4081640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-a con 0x7fe09c0a9d70 addr v1:172.21.15.117:6789/0 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-b con 0x7fe09c13f650 addr v1:172.21.15.173:6789/0 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient(hunting): picked mon.noname-c con 0x7fe09c1457f0 addr v1:172.21.15.117:6790/0 2024-03-29T11:29:53.737 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c142360 con 0x7fe09c0a9d70 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c145050 con 0x7fe09c13f650 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c141db0 con 0x7fe09c1457f0 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a4081640 10 monclient(hunting): authenticate will time out at 1462.049872s 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a161c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe09c13f650 0x7fe09c1d7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.117:34682/0 (socket says 172.21.15.117:34682) 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.740+0000 7fe0a161c640 1 -- 172.21.15.117:0/667325046 learned_addr learned my addr 172.21.15.117:0/667325046 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 client.?.objecter ms_handle_connect 0x7fe09c13f650 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 client.?.objecter ms_handle_connect 0x7fe09c1457f0 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375456582 0 0) 0x7fe09c145050 con 0x7fe09c13f650 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:53.738 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe080004eb0 con 0x7fe09c13f650 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175246508 0 0) 0x7fe09c141db0 con 0x7fe09c1457f0 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 10 monclient(hunting): my global_id is 4706 2024-03-29T11:29:53.739 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c145050 con 0x7fe09c1457f0 2024-03-29T11:29:53.741 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.741+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 187159760 0 0) 0x7fe09c145050 con 0x7fe09c1457f0 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe09c141db0 con 0x7fe09c1457f0 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe094001890 con 0x7fe09c1457f0 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 10 monclient(hunting): dump: 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:53.742 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1969089727 0 0) 0x7fe09c141db0 con 0x7fe09c1457f0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 >> v1:172.21.15.173:6789/0 conn(0x7fe09c13f650 legacy=0x7fe09c1d7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.742+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 >> v1:172.21.15.117:6789/0 conn(0x7fe09c0a9d70 legacy=0x7fe09c1d7470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 1 monclient: found mon.c 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c141f40 con 0x7fe09c1457f0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:53.743 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 5 monclient: authenticate success, global_id 4706 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 10 monclient: _renew_subs 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c142ac0 con 0x7fe09c1457f0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 10 monclient: _renew_subs 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c0ab800 con 0x7fe09c1457f0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0940019f0 con 0x7fe09c1457f0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0a4081640 1 librados: init done 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe0927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.743+0000 7fe093fff640 4 set_mon_vals no callback set 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.744+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe0940020c0 con 0x7fe09c1457f0 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.744+0000 7fe0927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:53.744 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.744+0000 7fe0927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.744+0000 7fe0927fc640 10 monclient: dump: 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.745+0000 7fe0a4081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.745+0000 7fe0a4081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.745+0000 7fe0a4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:53.745 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.745+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c142e60 con 0x7fe09c1457f0 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.745+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fe094019b00 con 0x7fe09c1457f0 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7fe094083af0 con 0x7fe09c1457f0 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0a1e1d640 10 monclient: get_auth_request con 0x7fe08006dc90 auth_method 0 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 10 client.4706.objecter ms_dispatch 0x7fe09c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-29T11:29:53.746 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.746+0000 7fe0927fc640 10 client.4706.objecter ms_handle_connect 0x7fe08006dc90 2024-03-29T11:29:53.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.754+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe09404f430 con 0x7fe09c1457f0 2024-03-29T11:29:53.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.754+0000 7fe0927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:53.747 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:53.754+0000 7fe0927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:54.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0a4081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0a4081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0a4081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.117:6790/0 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 --> v1:172.21.15.117:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe09c142510 con 0x7fe09c1457f0 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0927fc640 1 -- 172.21.15.117:0/667325046 <== mon.2 v1:172.21.15.117:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7fe094051360 con 0x7fe09c1457f0 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.056+0000 7fe0927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:29:54.049 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 10 librados: watch_flush enter 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 10 librados: watch_flush exit 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 10 client.4706.objecter successfully canceled tick 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 >> v1:172.21.15.117:6816/4276038175 conn(0x7fe08006dc90 legacy=0x7fe080070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 10 monclient: shutdown 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 >> v1:172.21.15.117:6790/0 conn(0x7fe09c1457f0 legacy=0x7fe09c1e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 shutdown_connections 2024-03-29T11:29:54.052 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 >> 172.21.15.117:0/667325046 conn(0x7fe09c066a00 msgr2=0x7fe09c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.060+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 shutdown_connections 2024-03-29T11:29:54.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.061+0000 7fe0a4081640 1 -- 172.21.15.117:0/667325046 wait complete. 2024-03-29T11:29:54.053 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:29:54.061+0000 7fe0a4081640 1 librados: shutdown 2024-03-29T11:29:54.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:00.876 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:01.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.027+0000 7fe43fd9d640 10 monclient: get_monmap_and_config 2024-03-29T11:30:01.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.027+0000 7fe43fd9d640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.019 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.027+0000 7fe43fd9d640 10 monclient: monmap: 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:30:01.028324+0000 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:30:01.028324+0000 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-b 2024-03-29T11:30:01.020 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.028+0000 7fe43fd9d640 1 Processor -- start 2024-03-29T11:30:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.028+0000 7fe43fd9d640 1 -- start start 2024-03-29T11:30:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.028+0000 7fe43fd9d640 10 monclient: init 2024-03-29T11:30:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.021 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4380b0770 addr v1:172.21.15.173:6789/0 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe43814a700 addr v1:172.21.15.117:6789/0 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4380ac5a0 addr v1:172.21.15.117:6790/0 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380aba70 con 0x7fe43814a700 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380aecb0 con 0x7fe4380b0770 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4380af150 con 0x7fe4380ac5a0 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.022 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43fd9d640 10 monclient(hunting): authenticate will time out at 1469.338610s 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43e33a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe4380ac5a0 0x7fe438151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47462/0 (socket says 172.21.15.117:47462) 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.029+0000 7fe43e33a640 1 -- 172.21.15.117:0/377552710 learned_addr learned my addr 172.21.15.117:0/377552710 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652253780 0 0) 0x7fe4380af150 con 0x7fe4380ac5a0 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): my global_id is 4709 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe424003520 con 0x7fe4380ac5a0 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715082985 0 0) 0x7fe4380aba70 con 0x7fe43814a700 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.023 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:30:01.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4380af150 con 0x7fe43814a700 2024-03-29T11:30:01.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1601357010 0 0) 0x7fe424003520 con 0x7fe4380ac5a0 2024-03-29T11:30:01.024 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.030+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 >> v1:172.21.15.173:6789/0 conn(0x7fe4380b0770 legacy=0x7fe438149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:01.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 >> v1:172.21.15.117:6789/0 conn(0x7fe43814a700 legacy=0x7fe4380abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.025 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 1 monclient: found mon.noname-b 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.117:6790/0 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 --> v1:172.21.15.117:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4380ac9d0 con 0x7fe4380ac5a0 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe434001690 con 0x7fe4380ac5a0 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43fd9d640 5 monclient: authenticate success, global_id 4709 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43fd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43cb37640 10 monclient: dump: 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.026 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.032+0000 7fe43fd9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:01.027 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.2 v1:172.21.15.117:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe434001870 con 0x7fe4380ac5a0 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 1 -- 172.21.15.117:0/377552710 <== mon.2 v1:172.21.15.117:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe434001cc0 con 0x7fe4380ac5a0 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43cb37640 10 monclient: dump: 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:30:01.028 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43fd9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:01.029 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.033+0000 7fe43fd9d640 4 set_mon_vals no callback set 2024-03-29T11:30:01.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 10 monclient: shutdown 2024-03-29T11:30:01.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:01.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 1 -- 172.21.15.117:0/377552710 >> v1:172.21.15.117:6790/0 conn(0x7fe4380ac5a0 legacy=0x7fe438151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.040 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 1 -- 172.21.15.117:0/377552710 shutdown_connections 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 1 -- 172.21.15.117:0/377552710 >> 172.21.15.117:0/377552710 conn(0x7fe43806b660 msgr2=0x7fe43806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 1 -- 172.21.15.117:0/377552710 shutdown_connections 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.034+0000 7fe43fd9d640 1 -- 172.21.15.117:0/377552710 wait complete. 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.035+0000 7fe43fd9d640 10 monclient: build_initial_monmap 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.035+0000 7fe43fd9d640 10 monclient: monmap: 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:epoch 0 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:last_changed 0.000000 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:created 0.000000 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.noname-a 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.noname-c 2024-03-29T11:30:01.041 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 librados: starting msgr at 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 librados: starting objecter 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 Processor -- start 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 -- start start 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 librados: setting wanted keys 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 1 librados: calling monclient init 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.036+0000 7fe43fd9d640 10 monclient: init 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:01.042 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4380ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4380b0770 addr v1:172.21.15.117:6789/0 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe43814a700 addr v1:172.21.15.117:6790/0 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4381deb90 con 0x7fe4380b0770 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe438128400 con 0x7fe4380ac5a0 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 1 -- --> v1:172.21.15.117:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4381239d0 con 0x7fe43814a700 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43fd9d640 10 monclient(hunting): authenticate will time out at 1469.346430s 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43e33a640 1 --1- >> v1:172.21.15.117:6790/0 conn(0x7fe43814a700 0x7fe4381e6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6790/0 says I am v1:172.21.15.117:47776/0 (socket says 172.21.15.117:47776) 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe43e33a640 1 -- 172.21.15.117:0/1332173635 learned_addr learned my addr 172.21.15.117:0/1332173635 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 client.?.objecter ms_handle_connect 0x7fe43814a700 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 client.?.objecter ms_handle_connect 0x7fe4380b0770 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:01.043 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161258278 0 0) 0x7fe4381deb90 con 0x7fe4380b0770 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c004eb0 con 0x7fe4380b0770 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.2 v1:172.21.15.117:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600168986 0 0) 0x7fe4381239d0 con 0x7fe43814a700 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 10 monclient(hunting): my global_id is 4712 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.037+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c0051d0 con 0x7fe43814a700 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3427488691 0 0) 0x7fe41c004eb0 con 0x7fe4380b0770 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4381239d0 con 0x7fe4380b0770 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe430001c10 con 0x7fe4380b0770 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.044 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): dump: 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.2 v1:172.21.15.117:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895509030 0 0) 0x7fe41c0051d0 con 0x7fe43814a700 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe41c005b60 con 0x7fe43814a700 2024-03-29T11:30:01.045 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.2 v1:172.21.15.117:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe434001b80 con 0x7fe43814a700 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 10 monclient(hunting): dump: 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:30:01.046 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669353193 0 0) 0x7fe4381239d0 con 0x7fe4380b0770 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 >> v1:172.21.15.117:6790/0 conn(0x7fe43814a700 legacy=0x7fe4381e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.038+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 >> v1:172.21.15.173:6789/0 conn(0x7fe4380ac5a0 legacy=0x7fe438126010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 1 monclient: found mon.a 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4381df730 con 0x7fe4380b0770 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe42e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 10 monclient: _renew_subs 2024-03-29T11:30:01.047 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4381df9e0 con 0x7fe4380b0770 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 10 monclient: _renew_subs 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe438126fb0 con 0x7fe4380b0770 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.041+0000 7fe43fd9d640 1 librados: init done 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe430003970 con 0x7fe4380b0770 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2308202164 0 0) 0x7fe430001c10 con 0x7fe4380b0770 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.042+0000 7fe42e7fc640 10 monclient: dump: 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:epoch 1 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:fsid 5b002eed-53a1-4f21-85da-94f1de320a49 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:last_changed 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.048 INFO:teuthology.misc.health.smithi117.stderr:created 2024-03-29T11:18:25.995512+0000 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:election_strategy: 1 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:0: v1:172.21.15.117:6789/0 mon.a 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2: v1:172.21.15.117:6790/0 mon.c 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr: 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.043+0000 7fe42ffff640 4 set_mon_vals no callback set 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.043+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1508230101 0 0) 0x7fe430019950 con 0x7fe4380b0770 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe43db39640 10 monclient: get_auth_request con 0x7fe41c06e4e0 auth_method 0 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2466827614 0 0) 0x7fe4300535d0 con 0x7fe4380b0770 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 10 client.4593.objecter ms_dispatch 0x7fe438000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe42e7fc640 10 client.4593.objecter ms_handle_connect 0x7fe41c06e4e0 2024-03-29T11:30:01.049 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.044+0000 7fe43fd9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.045+0000 7fe43fd9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.045+0000 7fe43fd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.045+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe420005080 con 0x7fe4380b0770 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.053+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe43004f870 con 0x7fe4380b0770 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.053+0000 7fe42e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:01.050 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.053+0000 7fe42e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:01.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.365+0000 7fe43fd9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:01.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.365+0000 7fe43fd9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:01.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.365+0000 7fe43fd9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.117:6789/0 2024-03-29T11:30:01.358 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.365+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe420005330 con 0x7fe4380b0770 2024-03-29T11:30:01.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.366+0000 7fe42e7fc640 1 -- 172.21.15.117:0/1332173635 <== mon.0 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2355007690) 0x7fe43004f870 con 0x7fe4380b0770 2024-03-29T11:30:01.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.366+0000 7fe42e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:01.359 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.366+0000 7fe42e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:01.359 INFO:teuthology.misc.health.smithi117.stdout: 2024-03-29T11:30:01.359 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":11,"fsid":"5b002eed-53a1-4f21-85da-94f1de320a49","created":"2024-03-29T11:19:04.150254+0000","modified":"2024-03-29T11:29:09.075828+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:09.075828+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7bc368d8-fe29-40fa-bf23-810bd2452126","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bfb91a80-d14b-490a-9f7d-4d1e06a97587","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ea86a613-bb1e-45a8-b681-6a589fa7dc6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b25cbe47-f48e-4b45-9abb-fa5ebf3e9519","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"582ed39e-64ed-42fb-90cf-5936b771756c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbe2fc3d-7406-4ebd-8354-d20d160ef309","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"12944eb6-17fc-42fa-a782-e5dbd4783762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b93c3c95-e26b-4ebe-b527-2271096e491d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 10 librados: watch_flush enter 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 10 librados: watch_flush exit 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 >> v1:172.21.15.117:6816/4276038175 conn(0x7fe41c06e4e0 legacy=0x7fe41c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 10 monclient: shutdown 2024-03-29T11:30:01.362 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.369+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 >> v1:172.21.15.117:6789/0 conn(0x7fe4380b0770 legacy=0x7fe4381df020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.370+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 shutdown_connections 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.370+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 >> 172.21.15.117:0/1332173635 conn(0x7fe43806b660 msgr2=0x7fe4380b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.370+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 shutdown_connections 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.371+0000 7fe43fd9d640 1 -- 172.21.15.117:0/1332173635 wait complete. 2024-03-29T11:30:01.363 INFO:teuthology.misc.health.smithi117.stderr:2024-03-29T11:30:01.371+0000 7fe43fd9d640 1 librados: shutdown 2024-03-29T11:30:01.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:01.378 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:01.380 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:01.380 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:01.380 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.415 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:01.415 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:01.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.489 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:01.489 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:01.489 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.559 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:01.559 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:01.560 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.634 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:01.634 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:01.634 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.679 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:01.680 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:01.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.719 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:01.719 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:01.719 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.785 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:01.785 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:01.785 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.861 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:01.861 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:01.861 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:01.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.886 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:01.886 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:01.886 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.926 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:01.926 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:01.927 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:01.927 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.938 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:01.939 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:01.939 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:01.991 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:01.991 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:01.991 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:02.002 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:02.002 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:02.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:02.156 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:30:02.157 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:05.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:30:05.011 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:05.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:30:05.139 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:05.188 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:30:05.188 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:05.255 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:30:05.255 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:08.030 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:30:08.030 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:08.085 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:30:08.085 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:08.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:30:08.143 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:08.201 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:08.203 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/data/mon.a.tgz 2024-03-29T11:30:08.203 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T11:30:08.221 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.2nT9aieqbR 2024-03-29T11:30:08.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2nT9aieqbR 2024-03-29T11:30:08.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.2nT9aieqbR 2024-03-29T11:30:08.445 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.2nT9aieqbR is 211KB 2024-03-29T11:30:08.539 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.2nT9aieqbR 2024-03-29T11:30:08.555 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/data/mon.c.tgz 2024-03-29T11:30:08.555 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-29T11:30:08.611 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.mfAclxJZpY 2024-03-29T11:30:08.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.mfAclxJZpY 2024-03-29T11:30:08.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.mfAclxJZpY 2024-03-29T11:30:08.799 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.mfAclxJZpY is 212KB 2024-03-29T11:30:08.893 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.mfAclxJZpY 2024-03-29T11:30:08.907 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/data/mon.b.tgz 2024-03-29T11:30:08.907 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-29T11:30:08.925 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.3ZRFl9CUu8 2024-03-29T11:30:08.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3ZRFl9CUu8 2024-03-29T11:30:09.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.3ZRFl9CUu8 2024-03-29T11:30:09.139 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.3ZRFl9CUu8 is 212KB 2024-03-29T11:30:09.232 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.3ZRFl9CUu8 2024-03-29T11:30:09.248 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:09.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:09.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:09.331 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:09.369 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:09.369 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:09.371 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi117/crash 2024-03-29T11:30:09.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:09.399 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi173/crash 2024-03-29T11:30:09.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:09.435 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:09.436 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:09.443 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:09.503 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35166.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35166.log.gz 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40393.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-29T11:30:09.504 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40198.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40393.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40801.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38674.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39301.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38674.log.gz 2024-03-29T11:30:09.505 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38754.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39535.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38754.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39535.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39535.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38793.log 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-03-29T11:30:09.506 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0%/var/log/ceph/ceph-client.admin.39691.log: -- replaced with /var/log/ceph/ceph-client.admin.38793.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39769.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41230.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35063.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40315.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28953.log 2024-03-29T11:30:09.507 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39769.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.tmp-client.admin.28953.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28953.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35063.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40549.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40918.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39262.log: gzip -5 93.5% --verbose -- -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:09.508 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.41347.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39145.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40159.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41269.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40471.log 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40159.log.gz 2024-03-29T11:30:09.509 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41542.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40471.log: /var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39808.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.1.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40471.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5/var/log/ceph/ceph-client.admin.39808.log: --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39067.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39808.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39964.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T11:30:09.510 INFO:teuthology.orchestra.run.smithi117.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39067.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41698.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41854.log 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-03-29T11:30:09.511 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39028.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41113.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41854.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41035.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39028.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40237.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39106.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2024-03-29T11:30:09.512 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39106.log: gzip -5 --verbose -- 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40237.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.40276.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41503.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39106.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40276.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41191.log: gzip -5 --verbose -- 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr: /var/log/ceph/ceph-client.admin.41386.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.y.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38635.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-03-29T11:30:09.513 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38635.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr: -- /var/log/ceph/ceph-client.admin.39340.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41581.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38635.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39340.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38950.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41776.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34772.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38950.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.34772.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38950.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41074.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41425.log 2024-03-29T11:30:09.514 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 86.5% --verbose --/var/log/ceph/ceph-client.admin.41737.log: /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.0.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41425.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39613.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39847.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39184.log: /var/log/ceph/ceph-client.admin.39847.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39847.log.gz 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-03-29T11:30:09.515 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.2.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-29T11:30:09.516 INFO:teuthology.orchestra.run.smithi117.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:09.517 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39379.log 2024-03-29T11:30:09.518 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39730.log 2024-03-29T11:30:09.518 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38596.log: /var/log/ceph/ceph-client.admin.39379.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-03-29T11:30:09.518 INFO:teuthology.orchestra.run.smithi117.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39379.log.gz 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34724.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34484.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34724.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34724.log.gz 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.34684.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34484.log.gz 2024-03-29T11:30:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34524.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34444.log: /var/log/ceph/ceph-client.admin.34524.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34524.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34604.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34604.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34564.log: 87.7% 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34604.log.gz 2024-03-29T11:30:09.524 INFO:teuthology.orchestra.run.smithi173.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-03-29T11:30:09.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34644.log 2024-03-29T11:30:09.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34644.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34644.log.gz 2024-03-29T11:30:09.525 INFO:teuthology.orchestra.run.smithi173.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:09.528 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40354.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40354.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39730.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40354.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.39457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.41152.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38872.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-03-29T11:30:09.529 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-03-29T11:30:09.530 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-03-29T11:30:09.530 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.40627.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-03-29T11:30:09.530 INFO:teuthology.orchestra.run.smithi117.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-03-29T11:30:09.530 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:09.530 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.38518.log: /var/log/ceph/ceph-mon.c.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-03-29T11:30:09.541 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:20.808 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:20.960 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:21.480 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:25.298 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:25.826 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:30:26.903 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:30:27.047 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:30:27.524 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:54.929 INFO:teuthology.orchestra.run.smithi117.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:30:59.567 INFO:teuthology.orchestra.run.smithi117.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:30:59.569 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-29T11:30:59.569 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m50.113s 2024-03-29T11:30:59.569 INFO:teuthology.orchestra.run.smithi117.stderr:user 2m39.005s 2024-03-29T11:30:59.569 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m5.463s 2024-03-29T11:31:18.953 INFO:teuthology.orchestra.run.smithi173.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:18.954 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T11:31:18.955 INFO:teuthology.orchestra.run.smithi173.stderr:real 1m9.458s 2024-03-29T11:31:18.955 INFO:teuthology.orchestra.run.smithi173.stderr:user 1m55.383s 2024-03-29T11:31:18.955 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m4.316s 2024-03-29T11:31:18.955 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:18.955 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi117/log 2024-03-29T11:31:18.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:31:41.835 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi173/log 2024-03-29T11:31:41.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:00.414 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:00.664 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=09e8434b9dc14020a84c4754a8475391 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:00.672 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:00.680 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:00.715 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:00.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:00.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:00.792 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:00.801 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:00.801 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:00.804 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:00.818 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:00.821 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-29T11:32:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +459us[ +459us] +/- 91ms 2024-03-29T11:32:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -110us[ -117us] +/- 54ms 2024-03-29T11:32:00.822 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -268us[ -268us] +/- 86ms 2024-03-29T11:32:00.823 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:00.825 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:00.826 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-29T11:32:00.826 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +463us[ +458us] +/- 91ms 2024-03-29T11:32:00.826 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -120us[ -125us] +/- 54ms 2024-03-29T11:32:00.826 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -261us[ -261us] +/- 86ms 2024-03-29T11:32:00.826 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:00.827 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:00.863 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:00.863 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:00.897 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:00.971 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:01.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:01.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:01.108 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:01.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:01.147 INFO:teuthology.task.internal:Duration was 1261.370141 seconds 2024-03-29T11:32:01.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:01.182 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:01.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:01.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:01.228 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:01.231 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:01.541 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:01.542 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-29T11:32:01.542 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:01.562 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T11:32:01.562 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:01.589 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:01.589 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:01.605 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:01.659 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:01.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:01.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:01.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:01.807 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:01.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:01.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:01.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:01.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:01.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:01.901 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-29T11:32:01.928 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-29T11:32:01.951 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:01.980 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:02.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:02.016 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:02.017 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi117 2024-03-29T11:32:02.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:02.062 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688/remote/smithi173 2024-03-29T11:32:02.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:02.097 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:02.098 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:02.100 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:02.147 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:02.157 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:02.157 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:02.182 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:02.182 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:02.185 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:02.197 INFO:teuthology.orchestra.run.smithi117.stdout: 265835 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:02.201 INFO:teuthology.orchestra.run.smithi173.stdout: 265955 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:02.202 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:02.212 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:02.290 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:02.311 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-29T11:32:02.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:02.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.796945', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQRrBRMAfHdhGld6YwrTI+gcrcuY4ZGQ6eZPfb+SEII/W1Ud8U9x3m9CUG8aEOrXCwN0BLNEH12AXsdQvZ+lnc='} 2024-03-29T11:32:02.342 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-29T11:32:02.342 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:02.369 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:06.798182', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKBnQWxnGUNHEHELviyjeTBjknWQL26n5SWHO5wnQbwINWwIuFxYgXK4piOozwC0FHhyobrZijV9TgSstsfCG2k='} 2024-03-29T11:32:02.399 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-29T11:32:02.400 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:02.427 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:02.428 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:02.448 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-29T11:32:02.449 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:02.461 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-29T11:32:02.475 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:02.476 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:02.507 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-29T11:32:06.461 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:06.508 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:06.576 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-29T11:32:06.620 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-29T11:32:10.579 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:10.622 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:10.691 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-29T11:32:10.734 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-29T11:32:10.792 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-29T11:32:14.738 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:14.852 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-29T11:32:14.953 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-29T11:32:15.057 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1261.3701412677765 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=09e8434b9dc14020a84c4754a8475391 status: fail success: false 2024-03-29T11:32:15.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:15.133 INFO:teuthology.run:FAIL